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1.0 


EXECUTIVE  SUMMARY 


1.1  INTRODUCTION 

The  U.S.  Air  Force  has  sponsored  the  development  of  standard  ergonomics  assessment 
methodology  guides  and  management  tools  that  will  be  integrated  into  the  AFOSH  Program. 
These  methodologies  and  tools  will  be  used  as  a  means  to  minimize  or  eliminate  work-related 
musculoskeletal  disorders  (WMSDs)  associated  with  routine  exposure  to  ergonomics  risk  factors 
at  Air  Force  installations. 

This  Report  on  Alpha  and  End-user  testing  describes  the  test  results  for  the  Level  I  Ergonomics 
Methodology  Guide  Supplement  for  Warehouse  and  Service  Areas  (W/S  Guide  Supplement). 

The  Guide  Supplement  is  an  addition  to  the  Maintenance  and  Inspection  Guide.  The  results  from 
testing  are  an  indication  of  the  effectiveness  with  which  base  Bioenvironmental  Engineering 
Flights  (BEF)  can. use  the  methodology  to  conduct  aggressive  task-based  problem  solving  in  an 
Ergonomics  Problem  Area  (EPRA). 


1.2  TESTING  PROCESS 

The  testing  process  was  conducted  in  two  primary  phases:  Alpha  testing  and  End-user  testing. 
During  Alpha  testing,  ergonomists  tested  the  Methodology.  During  the  End-user  testing,  Air 
Force  personnel  tested  the  Methodology.  The  Methodology  was  tested  for  usability,  reliability, 
and  validity.  End-user  testing  was  performed  to  ensure  that  users  would  be  able  to  apply  the 
Methodology  as  intended.  Reliability  testing  was  performed  to  determine  how  consistently  that 
application  of  the  Methodology  yielded  the  same  results.  Finally,  validity  testing  was  conducted 
to  measure  how  closely  the  results  from  experienced  ergonomists  matched  the  results  obtained 
by  Air  Force  personnel.  The  methods,  rationale,  and  results  of  the  testing  are  detailed  in  this 
report. 

The  testing  and  data  analysis  applied  to  the  W/S  Guide  Supplement  was  less  detailed  than  that 
which  was  performed  for  the  previous  Administrative  Guide  and  Maintenance  and  Inspection 
(M/I)  Guide.  Specifically,  the  following  items  were  either  not  conducted,  or  conducted  in  a 
substantially  different  fashion,  for  the  W/S  Guide  Supplement: 

•  A  comparison  to  a  “  gold  standard”  ergonomist  evaluation  using  independent  assessment 
methodologies  was  not  included  in  Alpha  or  End-user  testing. 

•  The  W/S  Guide  Supplement  End-user  testing  included  training  on  using  the  Guide,  but  did 
not  on  the  two-hour  briefing  on  ergonomics  principles  that  was  provided  with  previous  beta 
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testing. 

•  Only  eight  Air  Force  personnel  participated  in  End-user  testing,  instead  often  personnel 
proposed  in  the  Test  Plan. 

•  Only  five  VCR’s  with  monitors  were  available  for  use  by  the  eight  participants,  so  some  test 
subjects  chose  to  complete  the  process  after  watching  the  videos  rather  than  concurrently 
wilh  the  video. 

Pacific  Environmental  Services,  Inc.  (PES),  its  critical  subcontractor.  The  Joyce  Institute/a  unit 
of  Arthur  D.  Little,  Inc.  (TJI/ADL),  the  IERA/RSHE  Technical  Program  Manager  (TPM),  the 
IERA/RSHE  Technical  Consultant,  and  the  MAJCOM  Surveillance  Manager  (MSM),  all  agreed 
that  the  test  results  from  file  previous  guides  established  the  validity  of  the  checklist  and  pattern 
matching  methodology.  The  previous  testing  and  data  analysis  for  the  Administrative  and  M/I 
Guides  was  extensive.  It  was  agreed  that  repeating  this  level  of  detail  would  not  likely  result  in 
additional  information,  or  changes,  to  the  format  and  content  of  this  Guide  Supplement,  since  the 
methodology  used  in  the  W/S  Guide  Supplement  is  very  nearly  identical  to  the  previous  Guides. 
The  TPM,  IERA/RSHE  Technical  Consultant,  and  MSM  wanted  to  effectively  manage  costs  on 
this  Guide  Supplement  development.  To  this  end,  testing  for  this  Guide  Supplement  was  limited 
in  scope. 

There  is  an  essential  difference  between  the  End-user  test  and  previous  beta  rests.  The, purpose  of 
the  beta  test  was  to  verify  the  accuracy  of  the  methodology  when  used  by  the  target  user 
population.  The  purpose  of  the  End-user  test  was  to  verify  the  ability  of  the  target  population  to 
use  the  methodology.  Table  1 . 1  summarizes  the  differences  between  the  two  types  of  tests. 


13  PROJECT  RESULTS 

The  results  of  the  testing  process  provide  evidence  of  the  validity,  reliability,  and  practicality  of 
the  Methodology.  The  results  are  summarized  below. 

13.1  Reliability 


The  agreement  between  ergonomists  during  Alpha  testing  of  the  W/S  Guide  Supplement  (59%, 
Kappa  =  0.24)  was  nearly  identical  to  the  Alpha  test  agreement  from  the  M/I  Guide  (64%,  Kappa 
=  0.29).  This  is  consistent  with  expectations  since  the  checklist  tools  are  highly  similar. 

The  reliability  results  from  the  End-user  testing  did  not  meet  expectations.  The  agreement 
results  were  considerably  lower  than  those  obtained  during  W/S  Guide  Supplement  Alpha  testing 
and  the  M/I  Guide  beta  testing.  Post  hoc  investigation  of  these  results  suggests  that  a 
combination  of  factors  contributed  to  the  decreased  agreement  among  end-users  in  the  testing  of 
the  W/S  Guide  Supplement,  including: 
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The  amount  of  training  provided  for  the  W/S  Guide  Supplement  user  test  subjects  was  less 
than  that  provided  for  the  M/I  beta  test  subjects.  During  the  M/I  Guide  beta  test,  the 
ergonomist  provided  two  types  of  training:  instruction  on  th  z  process  required  to  participate 
in  the  testing;  and,  instruct  ion  in  ergonomics  principles  so  that  the  subjects  would  have  a 
more  comprehensive  understanding  of  the  significance  of  job  factors  and  corrective  actions. 
In  the  W/S  Guide  Supplement  User  test,  the  ergonomist  provided  die  type  of  training  ^ 
specified  in  the  protocol  for  the  End-user  test;  namely,  instruction  in  the  process.  Training  m 
ergonomics  principles  was  not  provided  since  this  was  an  End-user  test,  with  different 
objectives  from  the  beta  test  The  TJI/ADL  test  administrator,  who  has  conducted  various 
types  of  user  tests,  was  concerned  about  introducing  bias  into  the  process  and  therefore 
hesitated  to  provide  too  much  additional  information  during  die  test.  He  provided  neutral 
instructions. 


•  Although  there  were  five  VCR  monitors  available  for  the  eight  test  subjects,  many  times  the 

subjects  chose  not  to  view  the  videotape  while  completing  the  checklist  The  test 
administrator  was  hesitant  to  insist  because  he  did  not  want  to  introduce  bias.  In  previous 
beta  tests,  all  of  the  subjects  watched  the  videotapes  while  completing  checklists.  This  may 
have  been  the  result  of  a  combination  of  differences  in  the  administrators*  specific  verbal 
instructions  and  the  fact  that  there  were  fewer  VCR  monitors  than  test  subjects.  However, 
given  the  room  layout,  all  of  the  participants  did  have  access  to  monitors  and  could  have 
used  them  as  they  completed  checklists.  Completing  checklists  from  memory,  rather  than 
fiom  direct  observation,  could  have  a  considerable  negative  impact  on  the  consistency  of  the 
results. 


•  The  types  of  jobs  performed  in  the  scenarios  used  for  testing  the  W/S  Guide  Supplement 

often  consisted  of  fewer  tasks  than  the  M/I  Guide  scenarios.  This  is  due  to  the  nature  of 
many  service  jobs,  such  as  a  Dishwasher,  which  consist  of  a  singular  task  (dishwashing) 
performed  repetitively.  In  comparison,  many  maintenance  jobs  (such  as  jet  engine  repair) 
often  include  a  greater  variety  of  tasks  (wrenching/iatchetmg,  prying,  inspecting,  and 

twisting/tying).  Some  of  the  case  studies  in  previous  guides  have  also  addressed  broad, 

singular  task  jobs,  such  as  Wiring,  Asserably/Repair  -Benchwork,  and  Assembly/ 
Disassembly-Internal  Components.  The  nature  of  the  jobs  in  warehouse  and  services  areas 
resulted  in  a  greater  number  of  one  and  two  task  jobs.  This  supplement  continues  to  allow 
technicians  to  break  a  job  down  into  one  task  or  multiple  tasks,  depending  upon  which  is 
most  appropriate  for  the  situation.  An  examination  of  the  users’  job  evaluations  indicates 
That  many  of  the  users  broke  jobs  into  more  tasks  than  was  necessary,  based  on  the  scenario 
instructions.  Since  agreement  on  job  factor  identification  is  based  on  the  job  factor  being 
identified  within  the  same  task,  these  variations  in  task  selection  resulted  in  decreased 
agreement  on  job  factor  items. 


•  There  were  fewer  End-user  test  subjects  participating  in  the  test  than  planned.  This  reduced 
the  power  of  the  subsequent  analyses  and  made  definitive  conclusions  about  the  meaning  of 
the  results  more  difficult. 
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Table  1.1 

Comparison  between  Beta  Testing  and  End-user  Testing 


lL  .  |  IMUlfeSJ: 

W~  '  --  -  —  .9  "  '  -  “  -  : 

1  , 

Emphasis 

Verify  the  accuracy  of 
methodology  when  used  by 
target  user  population. 

Verify  usability  of  the 
methodology  when  used  by 
target  user  population.  The 
intention  was  a  quick 
verification  rather  than  the 
previous  detailed  studies. 

Case  Studies  Evaluated 

Thirty  scenarios  were 
evaluated  by  ten  subjects 
covering  approximately  35  of 
50  case  studies. 

Ten  scenarios  were  evaluated 
by  eight  subjects  covering  10 
of  the  20  case  studies. 

Validity  Testing 

* 

Results  obtained  during  Alpha 
and  beta  testing  were 
compared  against  an 
independent  “gold  standard” 
evaluation. 

Gold  standard  evaluation  not 
performed. 

Inter-rater  Reliability 

Kappa  valued  for  agreement 
on  each  job  factor  response. 

Same. 

Sensitivity  Testing 

The  scenarios  were  classified 
as  low,  medium,  and  high 
based  on  the  gold  standard 
evaluation.  Discriminant 
analysis  was  applied  to  beta 
test  checklist  results,  Alpha 
test  consensus  results  and  gold 
standard  results. 

Gold  standard  evaluation  not 
performed. 

Solution  Agreement 

The  ergonomists  selection  of 
corrective  actions  was 
compared  to  the  beta  test 
subjects  to  determine 
agreement. 

Same. 

End-user  testing 

Measurements  of  time  and 
subjective  responses. 

The  number  of  subjective 
responses  was  expanded  and 
comment  questions  were 
added  to  increase  the  amount  j 
of  usability  information 
obtained.  Field  trials  were 
called  for  in  the  Test  Plan,  but 
not  completed  due  to  time 
constraints  at  the  base  during 
the  testing. 
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As  a  result  of  these  factors,  PES  and  TJI/ADL  believe  that  the  results  of  the  End-user  test  do  not 
accurately  reflect  the  reliability  of  the  W/S  Guide  Supplement.  The  fact  that  the  results  of  the 
Alpha  test  are  nearly  identical  to  the  Alpha  test  results  from  the  M/I  Guide,  combined  with  the 
similarities  in  the  Level  I  Checklists  for  these  two  tools,  suggests  that  the  performance  of  this 
Guide  Supplement  is  similar  to  previous  guides. 

1.3.2  Validity 

All  of  the  job  factor  questions  were  supported  by  scientific  research.  The  list  of  job  factor 
questions  was  inclusive  of  the  Job  Factors  common  to  warehouse  and  service  tasks. 

•  The  overall  theoretical  framework  of  the  checklist  was  designed  using  all  of  the  elements  of 
the  assessment  tools  that  were  previously  validated.  The  checklist  framework  is  consistent 
with  the  methodologies  presented  in  the  Level  I  Methodology  Guides  for  Administrative 
Work  Areas  and  for  Maintenance  and  Inspection  Work  Areas. 

•  The  corrective  actions  selected  by  the  end-user  agreed  with  the  solutions  selected  by  a 
consensus  o'f  ergonomists  between  45%  and  47%  of  the  time.  This  was  lower  than  the 
agreement  rates  obtained  in  testing  previous  guides.  This  reduced  solution  agreement  is 
likely  the  result  of  decreased  inter-rater  agreement  on  the  Job  Factor  questions,  discussed  in 
Reliability  above.  The  selection  of  corrective  actions  in  the  case  studies  is  organized  by  Job 
Factors.  After  identifying  the  presence  of  a  Job  Factor,  the  user  references  this  Job  Factor 
within  the  Case  Study  to  identify  an  appropriate  corrective  action.  If  user  agreement  on  Job 
Factor  identification  was  reduced  because  of  changes  in  the  testing  procedures,  it  would  be 
anticipated  that  user  agreement  on  corrective  actions  would  also  be  reduced.  Since  the 
corrective  actions  are  organized  by  Job  Factors,  users  must  first  agree  on  the  identification  of 
job  factors  before  corrective  action  agreement  can  occur.  In  spite  of  the  reduced  agreement 
on  corrective  action  Selection  from  previous  testing,  the  current  results  suggest  that  the  W/S 
Guide  Supplement  assists  end-users  in  generating  solutions  that  experts  would  recommend. 

1.3.3  Practicality  and  Usability 

The  Methodology  Guide  Supplement  received  favorable  usability  comments  and  was  well 
accepted  by  BEF  technicians. 

•  The  usability  ratings  for  the  W/S  Guide  Supplement  are  consistent  with  previous  results. 

Each  question  rating  had  an  average  between  1.5  and  2.4  on  a  scale  from  1  to  5,  with  scores 
of  1  indicating  the  most  positive  response.  These  scores  indicate  that  participants  had  a 
generally  favorable  opinion  towards  the  Methodology  Guide  Supplement.  While  the  W/S 
Guide  Supplement  End-user  testing  had  an  expanded  usability  evaluation  component  from 
the  M/I  beta  testing,  the  results  are  similar.  The  ratings  for  the  M/I  beta  test  ranged  from  1 .8 
to  2.3. 
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•  The  W/S  Guide  Supplement  met  and  improved  on  the  “time  for  completion”  requirements 
established  by  the  Air  Force.  The  mean  time  for  completing  the  Level  I  Checklist  and 
scoring  process  was  23.0  minutes  with  a  standard  deviation  of  13.4  minutes.  The  mean  time 
for  identifying  and  selecting  control  measures  was  13.6  minutes  with  a  standard  deviation  of 
9. 1  minutes.  The  original  criteria  provided  by  the  Air  Force  was  one  to  two  hours  for  data 
collection  and  analysis  with  an  additional  one  to  two  hours  for  control  identification. 

1.4  CONCLUSIONS 

PES  and  TJI/ADL  do  not  consider  results  of  the  End-user  test  to  accurately  reflect  on  the 
reliability  of  the  W/S  Guide  Supplement.  The  fact  that  the  results  of  the  Alpha  test  are  nearly 
identical  to  the  Alpha  test  results  from  the  M/I  Guide,  combined  with  the  similarities  in  the  Level 
I  Checklists  for  these  two  tools,  suggests  that  the  performance  of  this  Guide  Supplement  is 
similar  to  previous  guides.  PES  and  TJI/ADL  believe  that  additional  End-user  testing  of  the  W/S 
Guide  Supplement  would  not  likely  add  appreciable  value  to  the  reliability  or  validity  of  the  W/S 
Guide  Supplement.  Beta  testing  of  previous  Guides  validates  both  the  checklist  design  and  the 
pattern  matching  process  that  are  used  in  the  W/S  Guide  Supplement. 
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2.0 


DEVELOPMENT  PROCESS  SUMMARY 


The  U.S.  Air  Force  has  sponsored  the  development  of  standard  ergonomics  assessment 
methodology  guides  and  management  tools  that  will  be  integrated  into  the  AFOSH  Program. 
These  methodologies  and  tools  will  be  used  as  a  means  to  minimize  or  eliminate  work-related 
musculoskeletal  disorders  (WMSDs)  associated  with  routine  exposure  to  ergonomics  risk  factors 
at  Air  Force  installations. 


2.1  USE  OF  THE  METHODOLOGY  IN  THE  TIERED  APPROACH  TO 

PROBLEM-SOLVING 

The  W/S  Guide  Supplement  Guide  Supplement  to  the  M/I  Methodology  Guide  was  developed  to 
provide  BEF  with  a  process  for  conducting  a  basic  ergonomics  assessment.  Through  use  of  a 
simple  pattern-matching  process,  the  supplement  identifies  realistic  controls  that  will  effectively 
minimize  or  eliminate  employee  exposure  to  ergonomics  hazards  in  jobs  in  warehousing  and 
service  areas,  including  such  areas  as  food  service,  commissary  tasks  and  patient  care. 

The  requirements  for  the  Methodology  design  were  specified  by  Headquarters  Air  Force  Materiel 
Command,  Office  of  the  Command  Surgeon  (HQ  AFMC/SGC)  and  IERA/RSHE  (formerly  DET 
1 ,  HSC/OEMO).  Both  organizations  desired  an  effective  and  efficient  analysis  and  problem¬ 
solving  process  that  could  be  applied  to  the  full  variety  of  Air  Force  warehousing  and  service 
work  areas.  The  Methodology  was  to  be  designed  to  reflect  the  technical  capabilities  of  a  BEF 
technician  with  only  two  to  three  years  of  experience.  In  addition,  the  process  was  to  place 
primary  focus  on  identifying  appropriate  controls.  Due  to  the  high  demands  already  placed  on 
BEF  personnel  and  the  potential  lack  of  ergonomics  expertise,  the  Air  Force  requested  that  a 
“pattern-matching”  process  be  created  that  would: 

•  minimize  the  time  requirements  for  assessment  and  control  identification;  and 

•  enable  the  Air  Force  to  benefit  from  the  expertise  of  ergonomists  who  have  had  years  of 
experience  in  addressing  ergonomic  hazards. 

The  Methodology  is  designed  to  enable  the  user,  primarily  through  visual  observations  and 
employee/supervisor  interviews,  to: 

•  identify  potentially  hazardous  tasks  within  a  shop  and  job; 

•  determine  if  the  content  of  the  job  and  task(s)  meet  established  ergonomics  (risk  factor 
exposure)  criteria; 
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*  determine  which  type(s)  of  additional  (Level  II)  analyses  may  be  used  if  further 
quantification  of  ergonomics  hazards  is  required;  and 

•  choose  from  a  menu  of  control  options  (both  short-  and  long-term)  which  when 
implemented,  minimize  the  risk  of  musculoskeletal  disorders  and  the  hazards  identified 

within  the  job  and  tasks. 

The  Methodology  was  designed  to  enable  the  user  to  complete  data  collection  and  analysis  on  a 
typical  job  in  one  to  two  hours,  and  complete  the  control  identification  and  summary  report  in  an 
additional  one  to  two  hours.  The  Methodology  includes  ease  studies  for  typical  warehousing  and 
service  area  tasks.  The  case  studies  serve  as  the  basis  for  the  pattern-matching  process  that  is 
used  to  “match”  the  hazards  identified  in  the  tasks  with  controls  that  will  reduce  employee 
exposure  to  those  hazards.  The  Methodology  identifies  metrics  that  can  be  used  to  evaluate  the 
impact  of  ergonomics  improvements  on  employee  health,  safety,  and  performance  (eg-,  quality, 
and  productivity).  A  detailed  description  of  the  pattern-matching  process  is  provided  in  the  W/S 
Guide  Supplement. 


22  SCIENTIFIC  BASIS  FOR  THE  METHODOLOGY  DESIGN 

A  literature  review  was  conducted  prior  to  the  development  of  the  Level  I  Methodology  Guides 
for  Administrative  and  M/I  work  areas.  This  literature  review  was  completed  to  identify  exiting 
methodologies  that  could  be  used  as  the  basis  for  the  Level  I  Methodology.  Initial  results  of  the 
review  indicated  that  comprehensive  ergonomics  analysis/problem-solving  methodologies,  using 
pattern-matching  as  the  basis  for  control  identification,  did  not  previously  exist 

As  a  result,  the  literature  review  was  targeted  to  identify  analysis  methods  upon  which  the  Level 
I  Ergonomics  Assessment  Checklist  could  be  based.  A  detailed  description  of  the  literature 
review  process  was  provided  in  the  research  report  [1]  from  the  initial  Methodology 
development. 

Prior  to  commencing  development  of  the  W S  Guide  Supplement,  TJI/ADL  ergonomists 
conducted  an  additional  literature  review.  The  purpose  of  this  literature  review  was  to: 

•  Identify  new  methodologies  for  checklist  assessment  and  determine  the  impact  on  the  Level 
I  Checklist  design; 

•  Examine  emerging  research  on  risk  factor  exposure  and  WMSD  exposure;  and 

•  Identify  analysis  methods  and  proposed  solutions  which  may  be  unique  to  warehouse  and 
service  tasks. 

The  additional  literature  review  provided  TJI/ADL  with  the  most  current  research,  particularly  on 
materials  handling  issues.  As  a  result  of  the  search,  TJI/ADL  was  able  to  provide  additional 
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justification  for  the  Risk  Factor  decisions  and  additional  references  for  the  Glossary.  The 
Bibliography  was  expanded  with  the  new  references  to  enable  the  users  to  review  the  most 
current  research  on  the  topic. 
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3.0 


BASIS  FOR  VALIDATING  THE  METHODOLOGY 


3.1  APPROACH 

Testing  of  the  Methodology  was  completed  in  two  major  steps:  Alpha  testing  and  End-user 
testing. 

The  purpose  of  Alpha  testing  was  to  provide  an  iterative  basis  for  Methodology  development. 
Experienced  ergonomists  served  as  test  subjects  to  enhance  the  usability  and  reliability  of  the 
Methodology  prior  to  subsequent  testing.  End-user  testing  examined  the  performance  of  the 
Methodology  in  a  controlled  environment  designed  to  be  as  similar  as  possible  to  actual  use. 
End-user  testing  used  inexperienced  BEF  technicians  to  complete  the  testing. 

A  total  of  ten  test  job  scenarios  were  developed  containing  video  and  text  information  for  typical 
warehousing  and  service  tasks  (see  Appendix  E).  The  tasks  for  the  Job  Scenarios  were  selected 
to  represent  a  variety  of  USAF  warehousing  and  service  tasks  and  a  variety  of  risk  factor 
exposures.  The  job  scenarios  are  representative  of  tasks  that  would  typically  be  considered  for 
ergonomics  intervention,  rather  than  all  warehouse  and  service  tasks.  As  a  result,  each  task  had 
some  exposure  to  most  ergonomics  hazards.  The  Job  Scenarios  presented  a  “  standardized” 
result  of  the  data  collection  process  to  ensure  that  each  person  using  the  Methodology  based  their 
answers  on  the  same  information.  In  actual  use,  the  data  collection  process  is  a  critical 
component  for  obtaining  results  with  the  Methodology.  The  potential  for  variations  within  the 
data  collection  for  each  Level  I  assessment  were  not  evaluated.  It  is  expected  that  training  and 
experience  will  result  in  consistently  accurate  data  collection.  These  same  test  job  scenarios 
were  used  for  both  Alpha  and  End-user  testing. 


3.2  METHOD 

3.2.1  Research  Design 

It  was  estimated  the  five  subjects  would  be  sufficient  for  the  Alpha  test  to  provide  a  preliminary 
evaluation  of  the  guide  and  to  provide  guidance  for  required  changes.  It  was  also  estimated  that 
ten  subjects  would  be  sufficient  to  demonstrate  the  utility  of  the  Methodology.  The  End-user 
testing  was  designed  to  use  ten  BEF  technicians  as  subjects.  Only  eight  technicians  were 
available  for  testing.  Each  subject  (five  in  Alpha  test,  eight  in  End-user  test)  evaluated  each  of 
the  ten  Job  Scenarios.  The  scenarios  were  administered  in  random  order  to  control  order  effects. 
This  was,  therefore,  a  repeated  measures  design,  with  multiple  dependent  measures.  The 
dependent  measures  collected  for  each  scenario  were:  the  responses  to  each  of  24  Job  Factor 
questions;  the  priority  scores  and  ratings  for  each  of  the  five  body  regions  (shoulder/neck, 
hand/wnst/arm,  back/torso,  legs/feet  and  head/eyes);  the  overall  priority  rating  for  the  job;  the 
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corrective  action  selections;  and,  the  time  required  to  complete  the  evaluation. 

3.2.2  Alpha  Testing 

Alpha  testing  refers  to  testing  performed  by  the  ergonomists  during  the  Guide  Supplement 
development  process. 

3.2.2.1  Subjects.  Five  ADL/TJI  ergonomists  participated  in  Alpha  testing.  These 
ergonomists  were  experienced  in  the  use  of  assessment  tools  and  ergonomics  checklists.  Each 
ergonomist  participating  in  the  Alpha  test  had  an  advanced  degree  in  the  field  and  at  least  five 
years  of  full-time  ergonomics  experience.  Four  of  the  ergonomists  were  Certified  Professional 
Ergonomists  (CPE). 

3.2.2.2  Procedure  and  Apparatus.  Each  of  the  ergonomists  was  provided  with  the 
following  materials: 

•  Ten  Written  Job  Scenarios  and  the  corresponding  video  tape; 

•  Level  I  Ergonomics  Assessment  Checklist  (draft  1)  and  Scoring  Summary; 

•  Corrective  Action  List  (part  of  Checklist  Scoring  Summary); 

•  Case  Study  Manual  (draft  1);  and, 

•  User’s  Instructions. 

The  ergonomists  reviewed  the  videotapes  and  their  responses  to  the  Ergonomics  Assessment 
Checklists.  Based  on  this  information,  the  ergonomists  selected  solutions  from  the  case  study 
manual.  Each  ergonomist  selected  solutions  from  the  case  study  manual' for  all  ten  jobs.  After 
this  was  completed,  one  ergonomist  led  a  focus  group  session  to  determine  enhancements  to  the 
case  study  manual. 

The  Level  I  Ergonomics  Assessment  Checklist  was  used  to  analyze  the  job.  The  Checklist 
Scoring  Summary  was  used  to  score  the  checklist  and  record  results  of  the  pattern-matching 
process  using  the  Case  Study  Problem-Solving  Matrices.  User’s  instructions  were  provided  to 
ensure  that  each  ergonomist  followed  the  same  procedure  for  applying  the  Methodology.  During 
a  group  meeting,  the  ergonomists  commented  on  the  usability  of  the  Methodology  components 
and  user’s  instructions.  A  second  draft  of  the  checklists  and  scenarios  was  developed  to  reflect 
these  comments.  Section  4. 1 .2  of  this  report  discusses  changes  to  the  checklists.  The  scenarios 
were  modified  to  reflect  the  tasks  shown  in  the  videotape,  rather  than  all  potential  tasks  that 
someone  doing  that  job  might  perform.  It  was  determined,  as  in  previous  Guide  testing,  that  there 
was  insufficient  information  to  accurately  answer  or  evaluate  the  environmental  questions.  Since 
these  questions  do  not  figure  in  the  scoring  and  are  not  reflected  in  the  Case  Studies,  no  analysis 
was  conducted  on  these  questions. 

3.2.3  End-user  Testing 

Subjects.  Eight  Air  Force  personnel  were  selected  to  participate  in  a  single-step 
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End-user  test.  These  personnel  were  selected  by  AFMC  to  “match”  the  targeted  end-user 
population:  BEF  technicians  with  two  to  three  years  of  experience.  Eight  personnel  were 
provided  and  the  test  was  conducted  at  Hill  AFB. 

Ten  subjects  were  originally  requested  for  the  End-user  testing.  The  rationale  for  selecting  ten  as 
the  appropriate  number  of  subjects  was  based  on  the  practicalities  of  conducting  the  research. 

The  time  frame  and  the  budget  for  this  project  permitted  only  ten  subjects'  data  to  be  collected 
and  analyzed.  Ten  subjects  have  been  sufficient  to  demonstrate  reliability  in  previous  guide 
testing.  While  the  impact  of  reducing  the  number  of  subjects  from  ten  to  eight  is  not  clear,  it  is 
unlikely  that  the  reduction  in  subjects  improved  the  inter-rater  agreement. 

3.2.3.2  Procedure  and  Apparatus.  Each  of  the  End-user  testers  was  provided  with  the 

following  materials: 

•  Ten  Written  Job  Scenarios  and  corresponding  video  tape; 

•  Level  I  Ergonomics  Assessment  Checklist  (draft  2)  and  Scoring  Summary; 

•  Corrective  Action  List  (part  of  Checklist  Scoring  Summary); 

•  Case  Study  Manual  (draft  2),  and; 

•  User’s  Instructions. 

The  testing  process  and  materials  provided  were  the  same  as  for  the  Alpha  test  with  the 
appropriate  revisions  to  the  methodology  from  the  Alpha  test  focus  group.  Each  End-user  test 
subject  followed  the  User’s  Instructions  to  apply  the  Methodology  to  each  of  the  Job  Scenarios. 
The  testing  process  was  completed  in  two  days.  Each  End-user  test  subject  was  also  asked  to 
record  the  amount  of  time  required  to  complete  both  the  Level  I  Ergonomics  Assessment 
Checklist/Checklist  Scoring  Summary  and  the  pattern-matching/control-identification  process  for 
each  Job  Scenario.  Each  participant  completed  a  survey  regarding  the  usability  of  the  Guide 
Supplement  at  the  end  of  the  End-user  test. 


3.3  DATA  ANALYSIS  AND  SCIENTIFIC  BASIS 

The  purpose  of  the  data  analysis  was  to  show  that  the  design  of  the  Level  I  Methodology  meets 
the  project  goals.  To  do  this,  it  was  necessary  to  demonstrate  that  the  Methodology  is  reliable, 
valid  and  practical.  These  concepts  are  easily  defined. 

A  scale  is  said  to  be  valid  if  it  measures  what  it  is  intended  to  measure.  An  ergonomics 
assessment  tool,  such  as  the  Level  I  Checklist,  should  measure  some  aspect  of  human  health, 
comfort  or  performance  in  order  to  be  a  valid  measure.  Reliability,  on  the  other  hand,  refers  to 
the  accuracy  and  the  repeatability  of  the  measurement  of  a  variable.  The  reliability  of  an 
instrument  is  the  foundation  for  the  other  concepts.  An  instrument  must  be  reliable  in  order  to  be 
valid,  since  one  can  not  be  sure  that  the  “  valid  results”  obtained  one  time  will  be  repeated. 
Practicality  identifies  the  usefulness  and  usability  of  an  instrument. 
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Reliability  testing  of  assessment  tools  generally  takes  one  of  two  forms:  test/re-test  reliability  and 
inter-rater  reliability.  Test/re-test  reliability  defines  how  -well  the  same  person  will  achieve  the 
cami*  results  using  a  tool  at  different  times.  Inter-rater  reliability  defines  how  well  different 
people  will  agree  on  the  results. 

Several  techniques  have  been  used  and  reported  for  inter-rater  reliability  testing  with  ergonomic 
assecgment  tools.  The  coefficient  of  variation  was  used  in  one  study  (Stetson  et  al,  1991  [2]) 
with  scores  of  less  than  20  percent  for  most  measures.  Multiple  regression  has  also  been  used  [2J 
with  a  finding  of  no  statistically  significant  differences  between  raters  supporting  reliability. 
Kemmlert  (1994)  [3]  found  weighted  averages  of  Kappa  ranging  from  0.24  to  0.44  and  a  percent 
agreement  often  above  70  percent  Keyserling,  Stetson,  Silverstein  and  Brouwer  (1 993)  [4]  used 
inter-rater  agreement  percentages  as  a  part  of  their  validation.  There  is  little  consensus  on  the 
best  methods  for  demonstrating  inter-observer  agreement  (Meister,  1985  [5]). 

Individual  questions  were  eval  uated  for  reliability  by  calculating  Kappa  statistics  for  the  Alpha 
and  End-user  tests.  The  Kappa  statistic  was  chosen  because  it  represents  an  accurate  method  for 
testing  reliability  while  controlling  for  the  effects  of  chance.  The  data  meet  all  of  the 
assumptions  for  Kappa  (Cohen,  1960  [6];  Brennan  &  Prediger,  1981  [7])  which  are: 

•  the  objects  are  categorized  and  independent; 

•  the  raters  operate  independently;  and 

•  the  categories  are  independent,  mutually  exclusive,  and  exhaustive. 

Since  it  was  expected  that  a  certain  amount  of  agreement  would  occur  by  chance,  like  having 
50%  correct  on  a  true/false  test,  the  Kappa  statistic  reports  agreement  after  chance  has  been 
removed.  A  Kappa  value  can  be  interpreted  as  a  percent  of  agreement,  for  instance  a  Kappa  of 
0.75  indicates  an  agreement  rate  of  75%  after  chance  has  been  removed. 

For  interpreting  the  Kappa  values  the  interpretations  shown  in  Table  3.1  were  used,  consistent 
with  those  suggested  by  Landis  and  Koch  (1977)  [8],  Kappa  values  were  considered  to  be 
statistically  significant  if  the  95%  Lower  Confidence  Interval  (LCI)  exceeded  0.0. 

The  scores  (item  level,  body  region  level,  and  risk  rating)  from  both  the  Alpha  and  End-user  test 
sessions  were  analyzed  independently  using  the  Kappa  statistic. 
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Table  3-1 

Kappa  Value  Interpretation 


Kappa 

Values 

Interpretation 

0.81  to  1.0 

0.61  to  0.80 
0.41  to  0.60 
0.21  to  0.40 

0.0  to  0.20 
<0.0 

Almost  Perfect 

Substantial 

Moderate 

Fair 

Slight 

Poor 

Hie  validity  of  an  instrument  refers  to  the  property  of  measuring  what  is  intended.  A  variety  of 
methods  have  been  used  to  establish  the  validity  of  ergonomic  assessment  procedures.  The  most 
common  types  of  instrument  validity  are: 


Predictive  validity'.  Predictive  validity  answers  the  question,  “Do  the  results  of  the 
instrument  accurately  predict  outcome  measures?”  Predictive  validity  is  also  referred  to  as 
criterion-related  validity.  In  the  evaluation  of  ergonomics  assessment  instruments,  predictive 
validity  has  been  suggested  through  comparisons  with  incidence  rate  [9]  and  with  discomfort 
ratings  [1 0].  For  instance,  an  assessment  instrument  that  predicts  incidence  or  discomfort 
rates  would  have  predictive  validity.  Any  predictive  validity  measures  for  simple 
ergonomics  assessment  techniques  are  subject  to  confounding  variables  that  could  greatly 
reduce  the  statistical  power.  Predictive  validity  was  not  assessed  for  any  of  the  three  Guides. 


•  Concurrent  validity :  Concurrent  validity  addresses  whether  two  independent  assessment 

methods  produce  similar  results.  The  rationale  underlying  concurrent  validity  is  that  if  two 
independent  techniques  produce  the  same  results,  the  likelihood  that  the  instrument  is 
measuring  the  intended  characteristic  is  increased.  In  the  evaluation  of  ergonomics 
assessment  instruments,  concurrent  validity  has  been  supported  through  comparing  expert 
ratings  on  two  different  techniques  [3]  and  by  comparing  novice  users  of  a  simple  technique 
with  expert  users  of  a  more  detailed  methodology  [4]. 


The  primary  method  for  assessing  Ihe  concurrent  validity  of  the  previous  guides  was  to 
compare  the  test  evaluations  (simple  technique)  with  the  Gold  Standard.  The  Gold  Standard 
refers  to  the  ergonomists  who  completed  an  independent  assessments  using  alternate 
assessment  methods,  of  the  jobs  in  the  Administrative  and  M/I  Guide  development  process. 
The  purpose  of  the  Gold  Standard  was  to  provide  an  indicator  of  concurrent  validity  and  to 
suggest  content  changes  to  the  methodology  during  Alpha  testing.  The  factors  which  were 
considered  when  selecting  the  Gold  Standard  ergonomists  were:  experience  in  industrial 
ergonomics  training,  workstation  adjustments  and  modifications,  and  equipment  design; 
first-hand  knowledge  of  Air  Force  jobs;  and  limited  involvement  in  the  Level  I  Guide 
development.  The  Gold  Standard  results  were  compared  to  the  Alpha  and  beta  results  as  part 
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of  the  testing  for  the  Administrative  and  M/I  Guides.  Due  to  the  similarities  between  the 
W/S  Guide  Supplement  and  M/I  Guides,  the  Gold  Standard  comparison  was  not 
incorporated  into  the  testing  design  for  the  W/S  Guide  Supplement. 

•  Content  validity :  Content  validity  refers  to  the  degree  to  which  the  items  used  in  an 

instrument  are  factors  that  are  considered  important  in  the  intended  characteristic.  Content 
validity  has  been  reported  as  an  item  match  with  scientific  literature  [3].  The  content  validity 
was  based  on  using  referenced  criteria  to  select  questions  during  tool  development. 
Kemmlert  [3]  reports  content  validity  for  the  PLIBEL  method.  Scientific  literature  was 
reviewed  to  provide  references  for  each  item  in  PLIBEL.  The  primary  approach  for 
assessing  content  validity  for  the  Level  I  Checklist  was  to  provide  support  in  the  literature 
(particularly  validated  assessment  tools)  for  all  Job  Factor  questions  and  scaling  approaches. 

While  the  practicality  of  the  Methodology  is  perhaps  the  most  important  consideration,  it  is  also 
the  most  difficult  to  quantify  in  a  short-term  test.  The  solid  test  of  practicality  is  how  frequently 
the  tool  is  applied  in  examining  workstations  and  the  results  obtained  through  the  changes.  In 
the  shorter-term,  several  measures  were  used  to  examine  the  practicality  of  the  tool.  The 
usability  comments  were  tabulated,  with  the  assumption  that  a  highly  useable  tool  is  more 
practical  than  other  tools.  The  descriptive  statistics  of  time  requirements  were  compared  to  the 
criteria  established,  with  the  assumption  that  an  effective  tool  that  can  be  completed  within  the 
time  suggestions  has  some  practicality.  The  overall  agreement  regarding  solution  options  was 
calculated  and  reported,  based  on  the  assumption  that  a  method  which  leads  to  consistent 
solution  recommendations  is  practical.  While  none  of  these  measures  proves  practicality, 
positive  results  in  all  these  areas  would  suggest  that  technicians  would  find  that  the  tool  is  a 
practical  one  to  use. 


3-6 


4.0 

RESULTS 

4.1 

RELIABILITY 

4.1.1 

Summary 

The  reliability  results  from  the  Alpha  test  of  the  W/S  Guide  Supplement  are  similar  to  the  results 
obtained  during  the  Alpha  testing  of  the  M/I  Guide.  The  average  agreements  were  both  near  60% 
(W/S  Guide  Supplement=59%,  M/I=64%).  The  mean  value  of  Kappa  for  individual  questions  is 
also  similar  between  the  test  groups  (W/S  Guide  Supplement  =  0.24,  M/I  =  0.29).  These  rates  are 
within  the  range  of  weighted  averages  of  Kappa  (0.24  to  0.44)  found  by  Kemmlert  [3].  These 
results  indicate  a  consistent  agreement  on  individual  question,  and  this  agreement  is  classified  in 
the  slight  to  fair  range.  The  similarity  in  results  between  the  two  guides  is  consistent  with 
TJI/ADL’s  expectations,  since  the  checklist  part  of  the  W/S  Guide  Supplement  is  nearly  identical 
to  the  M/I  Guide. 

Although  some  individual  questions  had  lower  agreement  during  the  Alpha  test  of  the  W/S  Guide 
Supplement  than  they  had  on  the  Alpha  test  of  the  M/I  Guide,  the  confidence  intervals  displayed 
considerable  overlap  in  most  cases.  It  was  also  the  case  that  many  questions  had  better  agreement 
during  the  W/S  Guide  Supplement  Guide  Supplement  Alpha  test  than  they  had  on  the  M/I  Guide 
Alpha  test,  still  with  overlapping  confidence  intervals.  The  most  plausible  explanation  is  that  the 
checklist  tool  (which  is  essentially  identical)  is  equally  effective  for  the  scenarios  used  in  the  W/S 
Guide  Supplement  and  the  M/I  Guide  Alpha  tests.  The  lower  number  of  task  scenarios  in  the  W/S 
Guide  Supplement  testing  (10  scenarios,  1 1  tasks)  than  in  the  M/I  Guide  testing  (30  scenarios,  70 
tasks)  reduced  the  power  of  the  test  and  resulted  in  larger  confidence  intervals  in  the  W/S  Guide 
Supplement  test  results.  As  a  result  it  would  be  risky  to  conclude  that  those  few  questions  that  had 
lower  scores  in  the  W/S  Guide  Supplement  test  than  the  M/I  Guide  test  were  the  result  of  small 
changes  in  the  checklist  tool  or  substantive  differences  between  W/S  Guide  Supplement  and  M/I 
Guide  task  characteristics.  It  would  be  equally  irresponsible  to  conclude  that  those  questions  that 
had  higher  scores  on  the  W/S  Guide  Supplement  test  than  the  M/I  Guide  test  indicated 
improvement  in  the  checklist  tool. 

Those  questions  that  had  a  Kappa  <  0.20  are  discussed  in  greater  detail  in  Table  4.2  in  order  to 
examine  the  potential  for  improvements  to  the  checklist  tool. 

The  reliability  results  from  the  End-user  testing,  however,  did  not  meet  TJI/ADL’s  expectations. 
The  agreement  results  were  considerably  lower  than  those  obtained  during  W/S  Guide  Supplement 
Alpha  testing  and  the  M/I  Guide  beta  testing.  Post  hoc  investigation  of  these  results  suggests  that  a 
combination  of  factors  contributed  to  the  decreased  agreement  among  end-users  in  the  testing  of 
the  W/S  Guide  Supplement,  including: 
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•  The  amount  of  training  provided  for  the  W/S  Guide  Supplement  user  test  subjects  was  less 
than  that  provided  for  the  M/I  beta  tester  subjects.  During  the  M/I  Guide  beta  test,  the 
ergonomist  provided  two  types  of  training:  instruction  on  the  process  required  to  participate  in 
the  testing;  and,  instruction  in  ergonomics  principles  so  that  the  subjects  would  have  a  more 
comprehensive  understanding  of  the  significance  of  job  factors  and  corrective  actions.  In  the 
W/S  Guide  Supplement  End-user  test,  the  ergonomist  provided  the  type  of  training  specified 
in  the  protocol  for  the  End-user  test;  namely,  instruction  in  the  process.  Training  in 
ergonomics  principles  was  not  provided  since  this  was  an  End-user  test,  with  different 
objectives  from  the  beta  test.  The  TJI/ADL  test  administrator,  who  has  conducted  various 
types  of  user  tests,  was  concerned  about  introducing  bias  into  the  process  and  therefore 
hesitated  to  provide  too  much  additional  information  during  the  test. 

•  Although  there  were  five  VCR  monitors  available  for  the  eight  test  subjects,  many  times  the 
subjects  chose  not  to  view  the  videotape  while  completing  the  checklist.  The  test  administrator 
was  hesitant  to  insist  because  he  did  not  want  to  introduce  bias.  In  previous  beta  tests,  all  of 
the  subjects  watched  the  videotapes  while  completing  checklists.  This  may  have  been  the 
result  of  a  combination  of  differences  in  the  administrators’  specific  verbal  instructions  and 
the  fact  that  there  were  fewer  VCR  monitors  than  test  subjects.  However,  given  the  room 
layout,  all  of  the  participants  did  have  access  to  monitors  and  could  have  used  them  as  they 
completed  checklists.  Completing  checklists  from  memory,  rather  than  from  direct 
observation,  could  have  a  considerable  negative  impact  on  the  consistency  of  the  results. 

•  The  types  of  jobs  performed  in  the  scenarios  used  for  testing  the  W/S  Guide  Supplement  often 
consisted  of  fewer  tasks  than  the  M/I  Guide  scenarios.  This  is  due  to  the  nature  of  many 
service  jobs,  such  as  a  Dishwasher,  which  consist  of  a  singular  task  (dishwashing)  performed 
repetitively.  In  comparison,  many  maintenance  jobs  (such  as  jet  engine  repair)  often  include  a 
greater  variety  of  tasks  (wrenching/ratcheting,  prying,  inspecting,  and  twisting/tying).  Some 
of  the  case  studies  in  previous  guides  have  also  addressed  broad,  singular  task  jobs,  such  as 
Wiring,  Assembly/Repair  -  Benchwork,  and  Assembly/  Disassembly-Internal  Components. 
The  nature  of  the  jobs  in  warehouse  and  services  areas  resulted  in  a  greater  number  of  one  and 
two  task  jobs.  This  supplement  continues  to  allow  technicians  to  break  a  job  down  into  one 
task  or  multiple  tasks,  depending  upon  which  is  most  appropriate  for  the  situation.  An 
examination  of  the  users’  job  evaluations  indicates  that  many  of  the  users  broke  jobs  into  more 
tasks  than  was  necessary,  based  on  the  scenario  instructions.  Since  agreement  on  job  factor 
identification  is  based  on  the  job  factor  being  identified  within  the  same  task,  these  variations 
in  task  selection  resulted  in  decreased  agreement  on  job  factor  items. 

•  There  were  fewer  End-user  test  subjects  participating  in  the  test  than  planned.  This  reduced  the 
power  of  the  subsequent  analyses  and  made  definitive  conclusions  about  the  meaning  of  the 
results  more  difficult. 

As  a  result  of  these  factors,  PES  and  TJI/ADL  believe  that  the  results  of  the  End-user  test  do  not 

accurately  reflect  the  reliability  of  the  W/S  Guide  Supplement.  The  fact  that  the  results  of  the 
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Alpha  test  are  nearly  identical  to  the  Alpha  test  results  from  the  M/I  Guide,  combined  with  the 
similarities  in  the  Level  I  Checklists  for  these  two  tools,  suggests  that  the  performance  of  this 
Guide  Supplement  is  similar  to  previous  guides. 


4.1.2  Task  Selection  Strategies 

End-users  selected  a  wide  range  of  tasks  to  include  in  the  analysis.  One  purpose  of  the  highly 
structured  scenarios  was  to  minimize  variation  at  the  task  selection  level.  This  was  not  the  result  in 
the  End-user  testing.  In  addition  to  including  tasks  that  were  not  listed  in  the  scenario  instructions, 
many  users  included  tasks  that  are  not  on  the  task  list.  For  instance,  in  this  testing  protocol  some  of 
the  variations  observed  on  the  meatpacking  scenario  were: 

•  Using  one  task:  meat  cutting  (which  directly  corresponds  to  the  scenario  instructions). 

•  Using  three  tasks:  cutting  meat,  wrapping  meat,  placing  meat  on  trays  (two  of  which  are  not 

on  the  task  list). 

* 

•  Using  three  tasks:  knife,  band  saw  and  unpacking  (none  of  which  are  on  the  task  list). 

•  Using  three  tasks:  lifting,  cutting  and  packaging. 

These  changes  in  task  selection  alter  the  task  duration  and  thus  the  score  assigned  to  a  Job  Factor 
Frequency  Rating.  Furthermore,  breaking  the  job  into  different  tasks  will  result  in  different  Job 
Factor  Frequency  Ratings.  For  instance,  assume  the  two  types  of  cutting  (knife  and  band  saw) 
account  for  40%  each  of  the  total  job  time.  Also  assume  that  the  job  factor  “bent  wrist”  occurs 
repeatedly  with  the  knife  but  not  with  the  band  saw.  If  these  are  combined  as  a  single  meat-cutting 
task,  consistent  with  the  task  description  in  the  case  studies,  a  Job  Factor  occurs  “Sometimes”  in  a 
task  with  “High”  Task  Frequency.  In  the  two  types  of  cutting,  however,  a  Job  Factor  occurred 
“Frequently”  in  a  task  with  “Moderate”  Task  Frequency.  The  greater  the  variation  in  task 
selection,  the  greater  the  variation  in  responses  and  scoring.  The  purpose  of  highly  structured 
scenarios  was  to  eliminate  this  potential  variation  from  the  test  scenario  structures  were  very 
effective  in  this  regard  in  previous  testing.  Training  and  experience  should  reduce  variation  in 
actual  use. 

The  structure  of  the  case  studies  and  task  list  may  have  contributed  to  this  increased  variation  in 
task  selection.  While  the  W/S  Guide  Supplement  does  not  abandon  the  concept  of  breaking  jobs 
into  component  tasks,  the  types  of  jobs  performed  in  W/S  Guide  Supplement  scenarios  often 
consisted  of  fewer  tasks  than  the  M/I  Guide  scenarios.  This  is  the  result  of  the  nature  of  many 
service  jobs  (such  as  a  dishwasher)  which  consist  of  a  singular  task  (dishwashing)  performed 
repetitively.  In  comparison,  many  maintenance  jobs  (such  as  jet  engine  repair)  often  include  a 
greater  variety  of  tasks  (wrenching/ratcheting,  prying,  inspecting,  and  twisting/tying). 
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The  discussion  regarding  the  distinction  between  task  and  job  has  been  an  on-going  one  during  this 
development  process.  Some  of  the  case  studies  in  previous  guides  have  also  addressed  broad, 
singular  task  jobs,  for  example,  Wiring,  Assembly/Repair  -  Bench  work,  and  Assembly/ 
Disassembly-Internal  Components.  The  nature  of  the  jobs  in  warehouse  and  services  areas  result  in 
a  greater  number  of  broad,  singular  task  jobs.  This  W/S  Guide  Supplement  continues  to  allow 
technicians  to  break  jobs  down  into  one  task  or  multiple  tasks,  depending  upon  which  is  most 
appropriate  for  the  situation.  During  the  testing  of  the  W/S  Guide  Supplement,  some  participants 
created  generic  task  types,  in  addition  to  the  specific  job  function,  to  describe  the  jobs.  As  a  result, 
participants  analyzed  very  different  sets  of  tasks. 

The  instructions  appear  straightforward  and  complete:  “identify  the  tasks  in  the  Specific  Tasks  and 
Times  and  write  the  task  names  in  the  Work  Content  Matrix.”  The  written  instructions  were  the 
same  instructions  provided  for  the  testing  of  the  M/I  Guide.  Post  hoc  investigation  revealed  that  the 
TJI/ADL  administrator  of  the  M/I  beta  testing  instructed  participants  to  directly  transfer  the  tasks 
listed  in  the  Specific  Task  and  Time  table  on  the  scenario  into  the  Work  Content  Matrix.  Similar 
verbal  instructions  were  not  provided  during  the  W/S  Guide  Supplement  End-user  test.  This  subtle 
addition  to  the  written  instructions  provided  may  have  decreased  the  likelihood  that  participants 
would  add  tasks  to  the  list  provided  in  prior  testing  sessions. 

4.1.3  Checklist  Completion  Methods 

The  lack  of  television  monitors,  discussed  in  Section  4.1.1,  for  End-user  testing  may  have  also 
contributed  this  style  of  checklist  completion.  Since  eight  test  subjects  shared  five  video  monitors, 
some  subjects  watched  the  video  and  then  completed  the  checklist  from  memory. 

Completing  a  checklist  from  memory  is  likely  to  be  less  accurate  than  completing  the  checklist 
“live.”  In  previous  test  sessions,  the  test  subjects  completed  the  checklist  while  watching  the 
video.  This  explains  the  inclusion  of  tasks  that  were  not  contained  on  the  video  (such  as  bagging 
in  the  Scanning  Groceries  example),  and  resulted  in  reduced  agreement  among  test  subjects. 

4.1.4  Job  Factor  Question  Agreement 

The  percent  agreement  and  Kappa  value  were  calculated  for  each  job  factor  question  on  the  Alpha 
and  End-user  tests.  The  percent  agreement  is  based  on  the  formula  for  determining  “percent 
observed”  within  the  generalized  Kappa  model  (Bartko  &  Carpenter  [11]).  A  generalized,  simple 
Kappa  appropriate  for  multiple  raters  and  response  categories  [1 1]  is  reported  for  item  agreement 
for  both  the  Alpha  and  End-user  tests.  The  95%  Upper  and  Lower  Confidence  Intervals  are 
reported  for  each  question.  The  response  rates  for  scored  responses  (scores  greater  than  zero)  are 
also  reported. 

The  general  level  of  agreement  between  ergonomists  using  the  Level  I  Checklist  is  presented  in 
Table  4. 1 .  The  agreement  rates,  not  corrected  for  chance,  for  the  Alpha  test  questions  ranged 
between  29%  and  71%  for  those  questions  with  sufficient  response  rates  of  risk  factor  presence.  A 
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minimum  of  ten  non-zero  responses  was  established  to  define  sufficient  response  rates  for  both  the 
Alpha  and  End-user  test.  In  the  Alpha  test,  five  questions  did  not  have  sufficient  response  rates  to 
meaningfully  interpret  (questions  7,  18, 22, 23  and  24).  Previously,  Kemmlert  [3]  found  agreement 
Kappa  values  in  the  fair  to  moderate  range  for  individual  questions.  Those  questions  which  did  not 
meet  this  level  of  agreement,  but  had  a  sufficient  number  of  responses,  are  shaded  in  Table  4.1. 
These  shaded  questions  indicate  those  items  that  are  potentially  problematic  in  the  W/S  Guide 
Supplement.  A  discussion  of  each  of  these  questions  is  presented  in  Table  4.2. 

The  ergonomists  determined  that  a  compelling  reason  for  changing  a  question  would  be  needed 
before  substantially  altering  the  question.  This  was  based  on  the  desire  to  keep  the  instrument 
compatible  with  the  M/I  Guide.  There  were  several  changes  made  to  this  Guide  Supplement  to 
address  the  issues  raised  by  the  Alpha  test.  The  ergonomists  reviewed  all  elements  of  the  Guide 
Supplement  and  made  the  following  revisions  after  Alpha  testing: 

•  Placed  an  <OR>  in  between  the  two  parts  of  Job  Factor  questions  1  and  16. 

•  Made  modifications  to  Job  Factor  pictures  for  checklist  questions:  1, 2,  3,  9, 11, 14, 15,  and 
24. 

•  Made  modifications  to  Job  Factor  examples  for  checklist  questions:  5, 8, 9, 10, 1 8, 19, 20  22, 
23,  and  24. 

•  Provided  numerous  picture  revisions  to  the  glossary. 


4-5 


Table  4.1 

Testing  of  the  Item  Agreement  Among  Ergonomists  for  Alpha  Test 


1 .  Reaching 


2.  Arm  Forces 


3.  High  speed  shoulder 
movement 


5.  Wrist  bent 

55% 

0.37 

0.23 

0.50 

6.  Finger  repetitions 

59% 

0.26 

0.02 

0.49 

7.  Single  finger 

74% 

-0.02 

-0.59 

0.55 

9.  High  speed  H\VA  or  HWA 


67% 


0.28 

0.13 

0.43 

0.29 

0.29 

0.09  I  0.65 
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Table  4.2 

Discussion  of  Job  Factor  Questions  with  Kappa  <  0.20 
W/S  Guide  Supplement  Alpha  Test  Results 


f  '  . 


There  was  an  82%  consensus  rate 
when  examining  simply  job  factor 
presence.  The  low  agreement  on 
this  question  appear  to  be 
primarily  related  to  rating  the 
duration  of  this  factor.  No  change 
was  recommended. 


Although  the  response  rate  on  this 
question  was  too  low  to 
accurately  interpret  the  results,  it 
was  included  in  the  table  to 
illustrate  the  number  of 
hand/wrist/arm  (HWA)  questions 
with  low  agreement.  In  spite  of 
having  lower  agreement  on  4  of 
the  7  HWA  questions,  the  W/S 
Guide  Supplement  test  had  higher 
agreement  on  the  W/S  Guide 
Supplement  Priority  Rating. 


0.24  0.10  There  was  a  73%  consensus  rate 
when  examining  simply  job  factor 
presence.  The  low  agreement  on 
this  question  appeared  to  be 
primarily  related  to  rating  the 
duration  of  this  factor.  The  degree 
of  overlap  in  the  confidence 
intervals  between  the  W/S  Guide 
Supplement  and  M/I  tests 
suggests  the  lack  of  a  real 
difference.  The  W/S  Guide 
Supplement  test  also  resulted  in 
improved  agreement  on  HWA 
ratings  suggesting  that  additional 
changes  may  not  be  needed.  No 
change  was  recommended. 


Table  4.2  continued 


it  Sfefe-i- 

jj 

pm 

ill 

9.  High  speed 
HWAorHWA 
vibration 

0.11 

0.33 

The  degree  of  overlap  in  the 
confidence  intervals  between  the 
W/S  Guide  Supplement  and  M/I 
tests  suggests  the  lack  of  a  real 
difference.  Subjects  may  have  had 
some  difficulty  interpreting  “high 
speed”  in  this  question  and  in 
question  14. 

10.  Hard  edges 

* 

0.09 

0.29 

0.12 

0.02 

The  degree  of  overlap  in  the 
confidence  intervals  between  the 
W/S  Guide  Supplement  and  M/I 
tests  suggests  the  lack  of  a  real 
difference.  The  W/S  Guide 
Supplement  test  also  resulted  in 
improved  agreement  on  HWA 
ratings  suggesting  that  additional 
changes  may  not  be  needed.  No 
change  was  recommended. 

12.  Back  bending 

0.18 

0.32 

0.22 

0.13 

i 

< 

i 

The  degree  of  overlap  in  the 
confidence  intervals  between  the 
W/S  Guide  Supplement  and  M/I 
tests  suggests  the  lack  of  a  real 
difference.  Based  on  discussions 
between  the  ergonomists,  it 
appeared  that  moderate  amounts 
of  sideways  bending  (#12)  and 
twisting  (#13)  are  often 
interchanged  when  scoring  the 
checklist.  This  resulted  in 
disagreement  at  the  item  level  and 
agreement  at  the  body  region 
level.  The  W/S  Guide  Supplement 
test  resulted  in  improved 
agreement  on  Back/Torso  (BT) 
ratings  suggesting  that  additional 
changes  may  not  be  needed.  No 
change  was  recommended. 
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Table  4.2  continued 


13.  Back  twisting 

* 

|[ '» 
1  C*;feTI' 

1  Si 

Based  on  discussions  between 
the  ergonomists,  it  appeared 
that  moderate  amounts  of 
sideways  bending  (#12)  and 
twisting  (#13)  are  often 
interchanged  when  scoring  the 
checklist.  This  results  in 
disagreement  at  the  item  level 
and  agreement  at  the  body 
region  level.  The  W/S  Guide 
Supplement  test  resulted  in 
improved  agreement  on  BT 
ratings  suggesting  additional 
changes  may  not  be  needed. 

No  change  was  recommended 
for  this  question. 

14.  High  speed 
back  movements 

0.07 

0.34 

0.36 

0.13 

The  degree  of  overlap  in  the 
confidence  intervals  between 
the  W/S  Guide  Supplement  and 
M/I  tests  suggests  the  lack  of  a 
real  difference.  Subjects  may 
have  had  some  difficulty 
interpreting  “high  speed”  in 
this  question  and  in  question  9. 
No  change  was  recommended. 
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Table  4.2  continued 
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15.  Static  back 
position 

* 

-0.01 

0.12 

0.21 

0.17 

One  ergonomist  scored  all 
standing  tasks  as  static  back 
postures,  inferring  that  all 
standing  tasks  are  stressful  on 
the  back.  This  interpretation  is 
beyond  the  wording  of  the 
question.  This  problem  was  not 
identified  previously  because 
this  ergonomist  did  not 
participant  in  previous  Alpha 
tests.  This  ergonomist  was  also 
the  least  experienced  in  the  use 
of  the  Level  I  assessments. 

When  this  ergonomist’s  data 
was  removed,  the  agreement 
was  consistent  with  previous 
test  results.  This  appears  to  be 
a  case  “  knowing  too  much” 
and  reading  this  knowledge 
into  questions.  This  error  is  not 
likely  to  be  repeated  by  end- 
users.  No  change  was  the 
question  is  recommended. 

20.  Exposure  to 
hard  edges 

-0.01 

0.16 

0.35 

0.12 

At  the  conclusion  of  Alpha 
testing  this  question  was 
modified  to  incorporate  hard 
floor  surfaces.  This  should  lead 
to  improved  agreement. 

The  job  factor  questions  were  also  evaluated  by  calculating  the  percentage  of  time  the  ergonomists 
reached  a  consensus  level  of  80%  on  responses.  Three  separate  measures  of  consensus  are 
included:  consensus  on  duration  (Never,  Occasionally,  Sometimes  or  Frequently);  consensus  on 
Job  Factor  occurrence;  and  consensus  on  Job  Factor  absence.  The  calculation  of  the  consensus  rate 
is  an  additional  measure  of  agreement  requested  by  the  IERA/RSHE  Technical  Consultant.  While 
there  is  no  single  agreed  upon  measure  of  agreement,  as  stated  previously,  both  methods  (Kappa 
and  consensus  rates)  Eire  accepted  approaches  to  reporting  agreement.  The  approximate  consensus 
rate  has  been  reported;  however,  since  the  request  for  certain  types  of  analysis  had  not  been 
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anticipated  when  the  Work  Plan  for  this  effort  was  approved,  the  data  input  was  not  structured  in 
such  a  way  to  allow  this  to  be  accomplished  quickly.  The  data  is  input  by  job  factor  score 
(0, 1,  3, 4, ...),  rather  than  by  duration  designator  (N,  O,  S  and  F). 

An  analysis  was  performed  to  indicate  the  relative  direction  this  data  may  take  by  treating  all  “O’s” 
as  “Never’s”.  It  should  be  noted  that  this  is  not  exactly  accurate,  but  should  be  indicative  of  the 
response  trends.  The  results  of  this  analysis  for  the  Alpha  test  are  presented  in  Table  4.3.  The 
average  overall  consensus  rate  across  all  job  factor  questions  for  ergonomists  was  53%.  This 
consensus  rate  improved  to  73%  (34%  for  job  factor  present,  and  39%  for  job  factor  absent)  when 
considering  only  the  presence  or  absence  of  job  factors. 

Table  4.4  shows  Kappa  results  for  the  End-user  tests.  The  overall  agreement  was  slight,  (raw 
agreement  47%,  mean  value  of  Kappa  =  0.1 1).  Only  four  questions  fell  in,  or  above,  the  range  of 
Kappa  values  found  in  the  checklist  study  by  Kemmlert  [3]. 

Table  4.5  presents  the  End-user  test  consensus  rates  for  each  job  factor  question.  The  same 
limitations  noted  in  the  Alpha  test  consensus  rate  calculations  apply  to  the  End-user  test.  Just  as  the 
Kappa  results  fof  the  end-users  was  lower  than  the  Alpha  test,  the  consensus  rate  was  also  less.  The 
average  overall  consensus  rate  across  all  job  factor  questions  for  end-users  was  24%.  This 
consensus  rate  improved  to  53%  when  considering  only  the  presence  or  absence  of  job  factors. 

4.1.5  Overall  Score  Agreement 

The  reliability  of  the  Level  I  Checklist  ratings  for  the  body  regions  and  overall  job  were  measured 
using  a  generalized  Kappa  statistic  [1 1],  the  coefficient  of  variance  and  the  rate  of  consensus.  The 
rate  of  consensus  is  reported  for  overall  agreement,  agreement  for  each  job  priority  rating  (High, 
Medium  and  Low),  and  agreement  that  job  priority  indicated  intervention  was  warranted  (either 
High  or  Medium).  The  Alpha  test  results  of  these  evaluations  are  presented  in  Tables  4.6, 4.7,  and 
4.8.  The  End-user  test  results  of  these  evaluation  are  presented  in  Tables  4.9, 4.10,  and  4.1 1. 

The  Kappa  results  from  the  W/S  Guide  Supplement  Alpha  test  are  presented  along  with  a  summary 
of  the  M/I  Guide  Alpha  test  results.  A  comparison  of  the  body  region  priority  rating  agreement 
obtained  during  W/S  Guide  Supplement  Alpha  testing  with  that  obtained  during  M/I  Guide  Alpha 
testing  indicates  highly  similar  results.  Although  the  Kappa  values  vary  between  the  two  testing 
sessions,  there  is  considerable  overlap  in  the  confidence  intervals  suggesting  that  the  most 
reasonable  conclusion  would  be  that  the  W/S  Guide  Supplement  and  M/I  Guide  Alpha  test  results 
were  similar.  One  might  cautiously  conclude  that  the  W/S  Guide  Supplement  Alpha  test  results 
suggested  an  improvement  over  the  M/I  Guide  results  based  on  improvement  in  three  of  the  five 
(hand/wrist,  back/torso  and  head/eyes)  body  regions  and  the  overall  job  rating,  particularly  since  it 
appears  intuitive  that  hand/wnst  and  back/torso  job  factors  would  be  more  clearly  evident  in  a  set 
of  jobs  that  included  warehousing,  cashiers  and  meat  cutting  operations.  Such  a  conclusion  should 
be  cautious  and  tentative,  based  on  the  degree  of  overlap  in  the  confidence  intervals. 
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Table  4.3 

Level  of  Consensus  (80%  Agreement)  on  Job  Factor  Questions 
W/S  Guide  Supplement  Alpha  Test  Results 


1  'W'-'r  i§  ,vwi,  ft  'Silhrk'  I 
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1 .  Reaching 


2.  Arm  Forces 


3.  High  speed  shoulder 
movement 


4.  Neck  bent 


5.  Wrist  bent 

6.  Finger  repetitions 


7.  Single  finger 


8.  Hand  grip  forces 


9.  High  speed  HWA  or 
HWA  vibration 


10.  Hard  edges 


1 1 .  Cold  temperatures 


12.  Back  bending 


13.  Back  twisting 


14.  High  speed  back 
movements 


15.  Static  back 
position 

16.  Back  forces 


17.  Pushing/Pulling 


18.  Whole  body 
vibration 


19.  Fixed  position 
standing 


20.  Exposure  to  hard 

edges _ 

21.  Awkward  leg 

postures _ 

22.  Awkward  foot 
postures 


23.  Light  levels 

24.  Intensive  staring 
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Table  4.4 

Testing  of  the  Item  Agreement  Among  End  Users 


<g 

BPS 

| 

lil* 

t:s 

•i 

I i 
t  (li®  1 

1 .  Reaching 

37% 

0.09 

-0.02 

0.20 

78 

2.  Arm  Forces 

36% 

0.06 

HS9 

0.16 

50 

3.  High  speed 
shoulder  movement 

42% 

0.01 

0.17 

37 

4.  Neck  bent 

31% 

0.18 

80 

5.  Wrist  bent 

37% 

81 

6.  Finger  repetitions 

58% 

0.11 

27 

7.  Single  finger 

61% 

0.10 

-0.15 

0.35 

25 

8.  Hand  grip  forces 

33% 

0.14 

0.07 

0.21 

69 

9.  High  speed  HWA 
or  HWA  vibration 

33% 

0.00 

-0.14 

0.13 

42 

10.  Hard  edges 

37% 

0.06 

-0.06 

6.18 

45 

1 1 .  Cold  temperatures 

61% 

0.35 

0.18 

0.51 

37 

12.  Back  bending 

30% 

0.04 

-0.03 

0.11 

67 

13.  Back  twisting 

31% 

0.11 

0.05 

0.18 

65 

14.  High  speed  back 
movements 

46% 

0.17 

0.06 

0.29 

47 

15.  Static  back 
position 

34% 

0.07 

-0.02 

0.15 

70 

16.  Back  forces 

IK22 

0.16 

0.04 

0.28 

46 

17.  Pushing/Pulling 

H 12 

0.13 

-0.10 

0.37 

28 

18.  Whole  body 
vibration 

92% 

0.04 

-0.89 

0.97 

4 

19.  Fixed  position 
standing 

49% 

0.28 

0.20 

0.37 

66 

20.  Exposure  to  hard 
edges 

34% 

0.10 

0.03 

0.18 

66 

21.  Awkward  leg 
postures 

56% 

0.21 

0.03 

0.38 

34 

22.  Awkward  foot 
postures 

78% 

-0.07 

-0.60 

0.47 

10 

23.  Light  levels 

63% 

0.23 

-0.01 

0.48 

27 

24.  Intensive  staring 

58% 

0.13 

-0.11 

0.37 

27 
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Table  4.5 

Level  of  Consensus  (80%  Agreement)  on  Job  Factor  Questions 
End-user  Test  Results 
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1.  Reaching 

0% 

82% 

0% 

2.  Ann  Forces 

9% 

36% 

9% 

3.  High  speed  shoulder 
movement 

18% 

0% 

18% 

4.  Neck  bent 

0% 

91% 

0% 

5.  Wrist  bent 

0% 

91% 

0% 

6.  Finger  repetitions 

27% 

9% 

27% 

7.  Single  finger 

45% 

0% 

45% 

8.  Hand  grip  forces 

0% 

45% 

0% 

9.  High  speed  HWA  or 
HWA  vibration 

0% 

0% 

0% 

10.  Hard  edges 

9% 

0% 

9% 

1 1 .  Cold  temperatures 

45% 

18% 

45% 

12.  Back  bending 

0% 

64% 

0% 

13.  Back  twisting 

0% 

45% 

0% 

14.  High  speed  back 
movements 

18% 

27% 

18% 

15.  Static  back  position 

0% 

55% 

0% 

16.  Back  forces 

27% 

27% 

27% 

17.  Pushing/Pulling 

45% 

0% 

45% 

18.  Whole  body  vibration 

91% 

0% 

91% 

19.  Fixed  position 
standing 

27% 

64% 

18% 

20.  Exposure  to  hard  edges 

9% 

45% 

0% 

21.  Awkward  leg  postures 

36% 

9% 

27% 

22.  Awkward  foot  postures 

91% 

91% 

23.  Light  levels 

36% 

9% 

36% 

24.  Intensive  staring 

45% 

9% 

45% 

4-14 


Table  4.6 

Agreement  (Kappa)  on  Body  Region  and  Overall  Job  Ratings 
Alpha  Test  Results 


g§§§ 

H 

m 

Job  SN 

0.47 

0.14 

-0.04 

0.32 

0.59 

0.30 

Job  HWA 

0.62 

0.40 

0.23 

0.57 

0.57 

0.27 

Job  BT 

0.53 

0.20 

-0.01 

0.42 

0.45 

0.14 

JobLF 

0.43 

0.00 

-0.19 

0.19 

0.55 

0.20 

Job  HE 

0.96 

0.73 

-0.15 

1.61 

0.74 

0.22 

Job  Overall 

0.75 

0.34 

-0.06 

0.75 

0.62 

0.18 

Table  4.7 

Coefficient  of  Variance  for  Body  Region  Scores  and  Job  Priority  Score 
By  Scenario  for  the  Alpha  Test 


H 

§§gg 

jHHH 

mu 

gggj 

mu 

m 

1 

Mean 

6.4 

7.2 

4.6 

3.6 

4.6 

8.2 

CFVAR 

0.422164 

0.22821773 

0.917175 

0.697217 

0.247864 

0.180883 

2 

Mean 

11 

11.4 

9 

3 

1 

13.2 

CFVAR 

0.272727 

0.19218335 

0.582672 

0.62361 

1.414214 

0.24781 

3 

Mean 

3.2 

2.6 

6.4 

1.5 

0 

6.4 

CFVAR 

0.40745 

0.79755544 

0.422164 

0.745356 

* 

0.422164 

4 

Mean 

4.2 

1.8 

2.4 

2.8 

0.8 

4.4 

CFVAR 

0.353152 

0.248452 

0.756913 

0.465657 

1.045825 

0.259131 

5 

Mean 

7.4 

9 

9.6 

4.6 

0.2 

10.8 

CFVAR 

0.280222 

0.27216553 

0.502813 

0.423774 

2.236068 

0.40572 

6 

Mean 

7.4 

8.8 

17 

4.8 

0 

17 

CFVAR 

0.311104 

0.38030001 

0.342997 

0.64885 

* 

0.342997 

7 

Mean 

7 

5.2 

5.8 

3 

0.2 

8.2 

CFVAR 

0.553283 

0.34401046 

0.970737 

0.235702 

2.236068 

0.534363 

8 

Mean 

6.8 

4.2 

8.8 

3.8 

0 

9.8 

CFVAR 

0.573341 

0.31043821 

0.697731 

0.220174 

* 

0.565383 

9 

Mean 

10.2 

13.8 

13.6 

4.8 

0.6 

15.4 

CFVAR 

0.127827 

0.07938008 

0.343312 

0.400737 

2.236068 

0.218282 

10 

Mean 

9 

8.4 

11.2 

5 

0.4 

12.2 

CFVAR 

0.260579 

0.23206653 

0.411102 

0.374166 

2.236068 

0.297802 
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•  Value  is  undefined  when  mean  =0 

Due  to  the  broad  confidence  interval,  the  apparent  improvement  in  agreement  on  the  overall  job 
rating  can  not  be  confirmed.  Practical  significance  suggests  that  the  reliability  agreement  is 
moderate,  with  chance  corrected  agreement  rates  between  0%  and  75%.  Table  4.8  presents  the  rate 
of  consensus  among  ergonomists  for  body  region  ratings  and  job  priority  ratings.  The  data  is 
organized  by  overall  consensus  (percentage  of  jobs  where  at  least  80%  of  ergonomists  assigned  the 
same  rating),  consensus  on  jobs  with  for  each  priority  ratings,  and  the  consensus  on  jobs  when 
medium  and  high  priority  ratings  were  pooled  together.  The  results  indicate  that  on  8  of  the  10  jobs 
the  ergonomists  reached  a  consensus  conclusion  on  job  priority  rating. 


Table  4.8 

Consensus  Rates  for  Body  Region  Scores  and  Job  Priority  Score 

Alpha  Test  Results 


.... 

-fed 

m 

yiSl 

Job  SN 

40% 

30% 

10% 

0% 

80% 

Job  HW 

50% 

30% 

10% 

10% 

70% 

Job  BT 

40% 

30% 

0% 

10% 

70% 

Job  LF  | 

20% 

0% 

0% 

20% 

30% 

Job  HE 

100% 

0% 

10% 

90% 

10% 

Job  Overall 

80% 

70% 

10% 

0% 

100% 

Table  4.9 

Agreement  (Kappa)  on  Body  Region  and  Overall  Job  Ratings 
End-user  Test  Results 


imm  . 

^  ( 

| 

BUS! 

Job  SN 

0.39 

-0.03 

-0.15 

0.10 

Job  HW 

0.41 

0.04 

-0.09 

0.17 

Job  BT 

0.38 

-0.02 

-0.17 

0.12 

Job  LF 

0.29 

-0.05 

-0.14 

0.04 

Job  HE 

0.54 

-0.03 

-0.29 

0.24 

Job  Overall 

0.47 

-0.01 

-0.23 

0.20 
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Table  4.10 

Coefficient  of  Variance  for  Body  Region  Scores  and  Job  Priority  Score 
By  Scenario  for  the  End-user  Test 


sSssmM- 

•m"'  @ 

l  ,  ' 

r,-. 

r-v 

'fmm 

m 

i  __ 

1 

Mean 

7.375 

10.5 

9.625 

4.125 

3.625 

13.25 

CFVAR 

0.54207 

0.41042 

0.62569 

0.79286 

0.5699 

0.35111 

2 

Mean 

9.125 

9.375 

8.875 

5 

1.5 

10.375 

CFVAR 

0.34871 

0.50335 

0.56639 

0.42762 

1.12687 

0.40873 

3 

Mean 

7.5 

9 

11.5 

4.75 

1.25 

12.75 

CFVAR 

0.54277 

0.3849 

0.4919 

0.35138 

1.26491 

0.39717 

4 

Mean 

5 

5 

4.25 

4.25 

1.875 

6.625 

CFVAR 

0.52372 

0.54511 

0.61293 

0.39272 

1.08321 

0.30121 

5 

Mean 

7.125 

11.375 

11.875 

5.25 

0.75 

13.5 

CFVAR 

0.60221 

0.61257 

0.63088 

0.75335 

1.98406 

0.52974 

6 

Mean 

8.625 

7.375 

12.625 

5.25 

0.625 

13.875 

CFVAR 

0.39165 

0.28934 

0.51843 

0.71088 

2.82843 

0.3519 

7 

Mean 

7.125 

9.5 

8.875 

6.625 

1.125 

12.25 

CFVAR 

0.52212 

0.63657 

0.50187 

0.41875 

1.45963 

0.3258 

8 

Mean 

9.25 

7.75 

12.125 

5.5 

0.875 

12.875 

CFVAR 

0.56839 

0.89595 

0.5371 

0.45584 

1.42448 

0.55922 

9 

Mean 

9.25 

13 

11.625 

6.75 

1.125 

14.875 

CFVAR 

0.43914 

0.39007 

0.59052 

0.39396 

1.20551 

0.43037 

10 

Mean 

8.5 

7.375 

12.125 

4.625 

0.75 

12.5 

CFVAR 

0.51474 

0.62741 

0.39291 

0.48948 

1.85164 

0.39656 
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Table  4.11 

Consensus  Rates  for  Body  Region  Scores  and  Job  Priority  Score 
End-user  Test  Results 
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Job  SN 

0% 

0% 

0% 

0% 

80% 

Job  HW 

0% 

0% 

0% 

0% 

50% 

Job  BT 

0% 

0% 

0% 

0% 
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The  end-user  bo'dy  part  priority  ratings  and  overall  job  ratings  reflect  the  lack  of  agreement 
observed  in  the  individual  job  factor  questions.  The  agreement  rate  was  47%,  (Kappa  =  -0.01, 
95%LCI  =-0.23, 95%UCI  =  0.20).  The  consensus  rates  observed  illustrate  a  similar  lack  of 
agreement.  The  lack  of  agreement  on  priority  ratings  is  likely  due  to  the  lack  of  agreement  on  task 
selections  and  job  factors. 

4.2  CONTENT  VALIDITY 

There  is  a  strong  scientific  basis  for  the  inclusion  of  all  the  job  factor  questions  present  in  this 
analysis.  All  job  factor  questions  were  derived  from  peer-reviewed  journals,  established  technical 
books,  or  proposed  standards.  Each  source  has  been  identified  as  a  potential  WMSD  risk  factor 
and/or  cause  of  localized  fatigue  and  discomfort.  The  source  documentation  is  provided  in  the 
Checklist  Glossary  contained  within  the  Guide  Supplement. 

The  list  of  job  factor  questions  has  been  judged  to  be  complete  for  warehousing  and  service  tasks. 
At  least  32  individual  analysis  methods  were  evaluated  as  a  part  of  the  literature  review  process. 
All  risk  factor-based  questions  identified  in  the  32  methods  were  considered  for  inclusion  in  order 
to  insure  that  the  list  of  job  factor  questions  was  complete. 

The  question/response  structure  (i.e.,  postural  deviations  or  forces  over  a  period  of  exposure)  is 
based  on  a  well-established  model  of  how  damage  accumulates.  Several  existing  analysis  methods 
[12,  2]  which  use  this  basic  model  have  been  validated. 
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4.3 


PRACTICALITY 


The  most  significant  aspect  of  the  entire  process  is  the  effectiveness  of  the  Guide  Supplement  in 
enabling  the  users  to  select  the  appropriate  corrective  actions.  The  measure  of  practicality  used  in 
testing  focused  on  whether  the  Methodology  could  be  applied  and  completed  within  the  time 
guidelines  established  by  the  Air  Force. 

4.3.1  Usability 

The  end-users  completed  an  usability  questionnaire  at  the  end  of  the  test  session.  The  usability 
questionnaire  consisted  of  20  questions  with  ratings  on  a  five  point  Likert  scale.  These  questions 
were  grouped  into  three  categories;  answering  the  checklist  questions,  using  matrices  to  identify 
solutions,  and  overall  use  of  the  Guide  Supplement.  The  questionnaire  is  contained  in  Appendix 
B.  Three  questions  on  the  scale  were  reversed  (for  example,  disagree  would  be  the  desired 
response  to  “The  tool  is  better  suited  to  an  ergonomist  than  a  non-ergonomist”).  In  order  to  ease 
interpretation,  the  scoring  of  the  reversed  questions  have  been  adjusted  to  match  the  other 
questions.  The  average  responses  for  each  question  fell  between  1.5  and  2.4,  indicating  that 
participants  agreed  with  each  usability  statement.  The  statements  that  scored  the  most  favorable 
were: 

•  The  scores  can  be  generated  quickly  and  easily, 

•  The  tool  helps  me  to  generate  more  solution  options  than  I  would  have  on  my  own, 

•  The  scores  can  be  clearly  interpreted,  and 

•  The  activities  performed  in  warehouse  and  service  areas  are  covered  in  the  tool. 

The  similarity  of  responses  for  each  question  suggests  the  responses  may  reflect  an  overall  attitude 
towards  the  Methodology,  rather  than  specific  information  about  individual  questions.  With  this  in 
mind,  it  indicates  that  participants  had  a  generally  favorable  opinion  towards  the  Methodology. 

4.3.2  Time  Requirements 

The  time  requirements  for  completing  the  analysis  and  for  identifying  and  selecting  control 
measures  were  calculated  based  on  the  End-user  test  sessions.  The  mean  time  for  completing  the 
Level  I  Checklist  and  scoring  process  was  23.0  minutes  with  a  standard  deviation  of  13.4  minutes. 
The  mean  time  for  identifying  and  selecting  control  measures  was  13.6  minutes  with  a  standard 
deviation  of  9. 1  minutes.  These  values  are  well  within  the  criteria  established,  indicating  that  this 
Methodology  can  be  used  quickly. 

These  times  are  within  a  standard  deviation  of  the  mean  times  obtained  during  Maintenance  and 
Inspection  Guide  testing  (Checklist  mean  =  16.6  minutes  with  a  standard  deviation  of  10.8 
minutes;  corrective  action  mean  =  12.0  minutes  with  a  standard  deviation  of  7.4  minutes).  This 
suggests  that  the  time  required  to  complete  an  assessment  using  the  W/S  Guide  Supplement  is 
similar  to  that  required  with  the  M/I  Guide. 
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4.3.3 


Selection  of  Corrective  Actions. 


The  agreement  on  the  selection  of  corrective  actions  was  examined  in  two  directions.  First,  the 
percentage  of  time  in  which  a  corrective  action  was  selected  by  an  End-user  was  compared  to  a 
consensus  of  ergonomists.  Second,  the  percentage  of  corrective  actions  selected  by  the  consensus 
of  ergonomists  was  also  selected  by  the  End-users.  It  was  determined  that  the  straightforward 
agreement  rates  between  the  consensus  of  ergonomists  and  end-users  would  results  in  an  inflated 
agreement  based  on  non-selection  of  corrective  actions,  since  most  actions  were  not  selected  for 
any  given  scenario.  The  “consensus  of  ergonomists”  in  this  and  previous  testing  is  represented  by 
two  factors: 

•  a  majority  of  TJI/ADL  ergonomists  agreeing  during  the  initial  Alpha  testing,  or 

•  a  minority  agreeing  in  Alpha  testing  supplemented  by  subsequent  discussions  to  produce  a 
majority  opinion. 

The  percentage  of  time  the  end-users  selected  corrective  actions  that  were  the  same  as  the 
consensus  was  45%.  This  indicates  that  of  100  corrective  actions  selected  by  end-users,  45  would 
have  also  been  selected  by  an  experienced  ergonomist.  While  this  is  below  the  61%  agreement 
rate  from  the  M/I  Guide,  it  is  better  than  would  be  expected  based  on  the  agreement  on  the  Job 
Factors. 

The  percentage  of  time  that  a  corrective  action  selected  by  the  consensus  of  ergonomists  was  also 
selected  by  the  End-user  test  subjects  was  47%.  This  indicates  that,  on  average,  whenever  the 
consensus  ergonomist  identified  a  solution,  roughly  four  out  of  eight  End-users  would  also  have 
identified  that  same  solution. 

In  the  testing  of  all  the  guides,  TJI/ADL  has  noticed  that  End-users  were  more  likely  to  select 
solutions  regardless  of  feasibility.  The  ergonomists  appear  more  selective  in  identifying  solutions. 
In  actual  use,  this  effect  would  likely  be  minimized  by  the  involvement  of  multiple  base  personnel 
working  to  implement  a  solution. 

The  agreement  on  solutions  was  likely  reduced  from  previous  testing  efforts  due  to  the  lower 
agreement  between  end-users  on  job  factor  identification.  Since  corrective  action  selection  is 
driven  by  the  identification  of  job  factor  presence  in  the  case  study  matrices,  a  reduction  in 
agreement  at  the  job  factor  level  would  be  likely  to  result  in  a  decreased  agreement  on  corrective 
actions. 
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5.0 


CONCLUSIONS 


The  results  of  the  validation  process  provide  evidence  of  the  validity,  reliability,  and  practicality  of 
the  Methodology.  The  results  are  summarized  below. 


5.1  VALIDITY 

All  of  the  job  factor  questions  were  supported  by  scientific  research.  The  list  of  job  factor  questions 
was  inclusive  of  the  Job  Factors  that  commonly  occur  in  warehousing  and  service  area  tasks. 

The  overall  theoretical  framework  of  the  checklist  was  a  logical  structure  used  in  assessment  tools 
that  have  been  validated.  The  checklist  framework  is  also  consistent  with  the  Level  I  Methodologies 
for  Administrative  and  M/I  Work  Areas. 

The  corrective  actions  selected  by  the  end-user  agreed  with  the  solutions  selected  by  a  consensus  of 
ergonomists  between  45%  and  47%  of  the  time.  This  is  lower  than  the  agreement  rates  obtained  in 
testing  previous  guides.  This  reduced  solution  agreement  is  likely  the  result  of  decrease  inter-rater 
agreement  on  the  Job  Factor  questions.  Since  the  corrective  actions  are  organized  by  Job  Factors, 
users  must  first  agree  on  the  identification  of  job  factors  before  corrective  action  agreement  can 
occur.  In  spite  of  the  reduced  agreement,  the  results  suggests  that  the  W/S  Guide  Supplement  assists 
end-users  in  generating  solutions  that  experts  would  recommend. 


5.2  RELIABILITY 

The  agreement  between  ergonomists  during  Alpha  testing  of  the  W/S  Guide  Supplement  (59%, 
Kappa  =  0.24)  was  nearly  identical  to  the  Alpha  test  agreement  from  the  M/I  Guide  (64%,  Kappa  = 
0.29).  This  is  consistent  with  expectations,  since  the  checklist  tools  are  highly  similar. 

The  reliability  results  from  the  End-user  testing  did  not  meet  expectations.  The  agreement  results 
were  considerably  lower  than  those  obtained  during  W/S  Guide  Supplement  Alpha  testing  and  the 
M/I  Guide  beta  testing  Post  hoc  investigation  of  these  results  suggests  that  a  combination  of  factors 
contributed  to  the  decreased  agreement  among  end-users  in  the  testing  of  the  W/S  Guide 
Supplement,  including: 

•  The  amount  of  training  provided  for  the  W/S  Guide  Supplement  End-users  was,  by  design,  less 
than  that  provided  for  the  M/I  Guide  beta  test  subjects; 

•  The  end  users  added  tasks  to  the  lists  provided  in  the  W/S  Guide  Supplement  case  studies  and 
test  scenario; 

•  There  were  fewer  end-users  participating  in  the  test  than  planned;  and 
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•  Many  times,  the  End-users  chose  not  to  view  the  videotape  while  completing  the  checklist. 

As  a  result  of  these  factors,  PES  and  TJI/ADL  does  not  consider  results  of  the  End-user  test  to 
accurately  reflect  on  the  reliability  of  the  W/S  Guide  Supplement.  The  fact  that  the  results  of  the 
Alpha  test  are  nearly  identical  to  the  Alpha  test  results  from  the  M/I  Guide,  combined  with  the 
similarities  in  the  Level  I  Checklists  for  these  two  tools,  suggests  that  the  performance  of  this 
Guide  Supplement  is  similar  to  previous  guides.  PES  and  TJI/ADL  believe  that  additional  End- 
user  testing  of  the  W/S  Guide  Supplement  would  not  likely  add  appreciable  value  to  the 
reliability  or  validity  of  the  W/S  Guide  Supplement.  Beta  testing  of  previous  Guides  validates 
both  the  checklist  design  and  the  pattern  matching  process  that  are  used  in  the  W/S  Guide 
Supplement. 


5.3  PRACTICALITY  AND  USABILITY 

The  Methodology  received  favorable  usability  comments  and  was  well  accepted  by  BEF  test 
subjects.  The  usability  ratings  for  the  W/S  Guide  Supplement  are  consistent  with  previous  results. 
Each  question  rating  had  an  average  between  1 .5  and  2.4  on  a  scale  from  1  to  5,  with  scores  of  1 
indicating  the  most  positive  response.  These  scores  indicate  that  participants  had  a  generally 
favorable  opinion  towards  the  Methodology.  While  the  W/S  Guide  Supplement  End-user  testing  had 
an  expanded  usability  evaluation  component  from  the  M/I  Guide  beta  testing,  the  results  are  similar. 
The  ratings  for  the  M/I  Guide  beta  test  ranged  from  1.8  to  2.3. 

The  W/S  Guide  Supplement  met  and  improved  on  the  “  time  for  completion”  requirements 
established  by  the  Air  Force.  The  mean  time  for  completing  the  Level  I  Checklist  and  scoring 
process  was  23.0  minutes  with  a  standard  deviation  of  13.4  minutes.  The  mean  time  for  identifying 
and  selecting  control  measures  was  13.6  minutes  with  a  standard  deviation  of  9.1  minutes.  The 
original  criteria  provided  by  the  Air  Force  was  1  to  2  hours  for  data  collection  and  analysis,  with  an 
additional  1  to  2  hours  for  control  identification. 


5.4  SUMMARY 

In  summary,  the  primary  strengths  of  the  Guide  are: 

•  it  is  easy  to  use; 

•  it  has  shown  good  acceptance  by  the  most  likely  end-user  population  (BEF  technicians); 

•  it  meets  and  improves  on  the  “time  for  completion”  requirements  established  by  the  Air  Force; 
and 
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Biography 


Richard  Barker 

Certified  Professional  Ergonomist  (CPE) 
Arthur  D.  Little 


Mr.  Barker,  MA,  CPE  is  a  manager  with  Arthur  D.  Little,  within  the  Environmental,  Health,  and 

Safety  Division.  Mr.  Barker  has  been  an  ergonomics  consultant  since  1991. 

Assembly  and  Manufacturing 

•  For  an  United  States  Air  Force  depot  maintenance  facility,  Mr.  Barker  performed  an 
ergonomics  assessment  of  30  engine  and  engine  component  repair  workstations.  Based  on  the 
risk  factors  identified,  Mr.  Barker  proposed  a  range  of  corrective  actions  for  each 
workstation.  After  workstation  changes  had  been  implemented,  he  returned  to  document  the 
health,  quality  and  productivity  benefits  associated  with  the  ergonomic  improvements.  Mr. 
Barker  trained  personnel  at  this  base  to  perform  basic  and  advanced  ergonomic  assessments. 

(1996-1998) 

•  For  GE  Aircraft  Engine,  Mr.  Barker  conducted  training  courses  in  ergonomics  for  design 
engineers  in  Support  Equipment  Operations.  These  courses  focused  on  the  practical 
application  of  ergonomics  guidelines  to  the  design  of  fixtures  and  tooling.  In  the  test  engine 
development  area,  Mr.  Barker  performed  an  ergonomics  assessment  of  300  carts,  classified 
into  over  100  types  and  variants.  He  identified  a  priority  for  making  handle  design  and  wheel 
selection  interventions.  For  those  carts  requiring  new  handle  designs,  Mr.  Barker  identified 
design  concepts  and  worked  with  other  engineers  to  develop  design  drawings  and  prototypes. 
(1997) 

«  For  a  manufacturer  of  hydraulics  products,  Mr.  Barker  provided  ergonomics  assessment  and 
training  services.  Mr.  Barker  performed  an  overview  assessment  of  the  entire  operation 
through  a  combination  of  injury/illness  records  review,  employee  interviews  and  direct 
observation.  Based  on  this  information,  he  identified  work  areas  with  both  high  pnonty  and 
high  opportunity  for  intervention  for  additional  assessment  and  recommendation 
development.  Mr.  Barker  developed  short-term  and  long-term  recommendations  for 
workstation  improvement  for  each  job  identified.  Mr.  Barker  provided  training  for  plant 
personnel  in  the  identification  of  ergonomics  ha2ard$.  (Sun  Hydraulics,  1997) 

•  For  a  manufacturer  of  commercial  fuse  and  electric  products,  Mr.  Barker  has  conducted  an 
ergonomics  evaluation  of  one  fiise  assembly  and  packaging  line.  As  a  result  of  this 
evaluation,  he  identified  changes  in  the  workstation  designs  that  could  reduce  exposure  to 
ergonomics  hazards.  (S  &  C  Electric,  1997) 
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•  For  a  manufacturer  of  rice  cakes,  Mr,  Barker  provided  process  and  tool  modification 

recommendations  to  reduce  exposure  to  ergonomics  hazards  in  the  cake  popping  molds  area. 
These  recommendations  included  specific  product  suggestions  for  off-the-shelf  products  and 
detailed  design  specifications  for  custom  manufactured  tools.  Mr.  Barker  met  with  contract 
tool  engineers  to  discuss  the  suggested  modifications.  (Quaker  Rice  Cakes,  1996) 

•  For  the  machining  and  assembly  divisions  of  an  axle,  brake  and  transmission  manufacturer; 
Mr.  Barker  analyzed  workstations  and  ranked  them  according  the  potential  for  ergonomics 
injury.  He  then  made  workstation  design,  tooling  selection  and  process  change 
recommendations  to  reduce  the  risk  of  injury.  He  provided  ongoing  support  to  project 
engineers  in  the  workstation  redesign,  cost  justification  and  implementation  processes.  Mr. 
Barker  conducted  training  at  multiple  facilities  within  the  automotive  products  division.  He 
held  sessions  for  managers  outlining  the  business  case  for  ergonomics  intervention  within 
their  facilities.  Engineers,  safety  personnel,  medical  personnel,  and  members  of  the 
ergonomics  steering  committee  were  trained  in  hazard  identification,  risk  prioritization, 
solution  development,  and  solution  evaluation.  Area  teams  composed  of  production 
employees  coordinated  by  the  area  engineers  were  trained  to  identify  ergonomic  risks  and 
propose  low-cost  solutions.  ( 1 992-1 994) 


Mr.  Barker  reviewed  the  ergonomics  program  for  a  motorcycle  engine  assembly  facility. 
Based  on  this  program  assessment,  he  suggested  specific  changes  to  the  written  ergonomics 
plan  and  implemented  a  plant-wide  ergonomics  training  strategy.  Mr,  Barker  worked  with 
client  representatives  to  develop  site  specific  training  materials  and  risk  assessment  tools. 
The  resulting  training  course  is  required  for  all  incoming  manufacturing  engineers, 
supervisors,  union  safety  representatives  and  union  stewards.  (Harley  Davidson,  1994-1998) 


•  Mr.  Barker  trained  joint  management-union  ergonomics  committees  at  over  30  automotive 
assembly  and  components  facilities  in  ergonomics  hazard  evaluation  and  in  the  development 
of  injury  prevention  strategies.  As  part  of  these  site-specific  training  sessions,  Mr.  Barker 
conducted  ergonomic  hazard  assessments  and  developed  corrective  action  recommendations 
at  each  facility  to  serve  as  models  for  course  participants.  (1992-1994) 

•  Mr.  Barker  provided  ergonomics  design  support  to  the  engineering  department  for  several 
foundries.  He  recommended  short-  and  long-term  alternatives  for  both  current  operations  and 
proposed  new  areas.  The  potential  reduction  in  ergonomic  risk  was  quantified  for  each 
recommendation.  (Ford,  Chrysler,  GM,  1991-1994) 


•  For  a  manufacturer  of  extruded  plastic  automotive  components,  Mr.  Barker  assessed  the 
potential  impacts  of  job  changes  proposed  by  plant  engineers.  He  analyzed  jobs  to  detennine 
the  potential  for  retum-to-work  or  alternate  duty  classification.  (1993) 

*  For  a  manufacturer  of  truck  engines,  Mr.  Barker  reviewed  the  assembly  operations  to  identify 
ergonomics  hazards  and  provide  recommendations.  He  also  trained  the  joint  management- 
union  ergonomic  committee  in  hazard  identification  and  recommendation  development. 
(Navistar,  1995) 
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•  For  a  medical  products  manufacturer,  Mr.  Barker  assisted  plant  and  corporate  engineers  in 
the  clean  sheet  design  of  a  facility  by  analyzing  current  operations  and  developing  design 
alternatives.  The  project  included  workstation  design,  plant  layout,  part  transfer  and 
manpower  loading  components.  Mr.  Barker  also  provided  recommendations  for  workstation 
modifications  at  additional  facilities  within  the  U.S.,  U.K.  and  Mexico.  (1991-1994). 


Mr.  Barker  has  conducted  site  specific  training  for  engineers  and  members  of  ergonomics 
committees  focusing  on  hazard  identification,  solution  development  and  ergonomics 
committee  structure  for  a  wide  variety  of  industries  including  defense  contractors,  paper 
products,  lumber  and  wood  pulp  products,  beverage  bottling,  food,  chemical,  rubber  an 
tire,  steel,  electronics,  plastics,  fiberglass  and  automotive  parts,  (including  Freighthner,  GE 
Plastics,  Lukens,  Motorola,  Quaker  Oats  Company,  Seagram,  Yokohama,  1991-199/) 


Forensics 

•  In  an  arbitration  involving  a  union  claim  that  the  employer  was  staffing  at  unsafe  and 
unreasonable  levels,  Mr.  Barker  performed  assessments  to  identify  the  frequency  of 
ergonomics  stressors.  These  stressors  were  compared  to  guidelines  drawn  from  the 
research  literature  in  order  to  identify  the  impact  that  staffing  level  and  work  rates 
might  be  having  on  the  employees.  (1997) 

.  In  a  case  filed  under  the  Americans  with  Disabilities  Act,  Mr.  Barker  compared  the 
physical  demands  associated  with  the  job  to  the  work  restrictions  and  capabilities  of 
the  plaintiff.  (1997) 

•  In  response  to  an  OSHA  investigation,  a  manufacturer  of  hand-made  specialty 
products  requested  an  independent  ergonomics  assessment  and  recommendations  for 
workstation  and  process  improvement.  Mr.  Barker  performed  the  assessment, 
developed  recommendations  and  prepared  a  report  documenting  the  findings.  (1995) 

•  In  support  of  a  Cal-OSHA  abatement  process,  Mr.  Barker  has  provided  work  station  design 
recommendations  for  a  fruit  packaging  operation.  (1 993) 

•  In  support  of  an  OSHA  abatement  agreement  at  a  meatpacking  facility ,  Mr.  Barker  provided 
evaluations  and  redesign  recommendations.  Mr.  Barker  also  assessed  work  areas  for  potential 
alternate  duty  or  retum-to-work  placement.  (1992) 

•  In  support  of  an  OSHA  abatement  agreement  for  an  automotive  manufacturer,  Mr.  Barker 
provided  worksite  assessments  and  ergonomics  training  for  joint  union-management 
committees.  (1991-1993) 
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Packaging  end  Shipping 

•  For  a  manufacturer  of  cartridge  razors,  Mr.  Barker  performed  a  review  of  3  proposed 
packaging  lines.  This  review  included  the  identification  of  likely  ergonomics  hazards  and 
proposed  recommendations.  He  performed  the  review  based  on  blueprint  drawings,  sample 
product  packages  and  observation  of  similar  packaging  lines.  (Gillette,  1997-1998) 

•  For  a  manufacturer  of  cabinets,  Mr,  Barker  suggested  workstation,  equipment  and  process 
improvements  for  the  palletizing  and  loading  truck  operations.  (Fisher  Scientific,  1996) 

•  For  a  manufacturer  of  cosmetics,  Mr.  Barker  developed  design  guidelines  for  inspection,  card 
loading  and  cartoning  workstations.  These  design  guidelines  were  incorporated  into  training 
courses  for  the  company’s  process  and  manufacturing  engineers,  Mr,  Barker  also  prepared  an 
ergonomics  assessment  and  suggested  workstation  improvements  for  a  pick  tunnel  operation, 
in  this  facility’s  shipping  area.  (1994) 

•  For  a  facility  which  warehouses  oil  and  lubrication  products,  Mr.  Barker  identified 
workstation  improvements  for  the  product  picking  operations.  He  quantified  the  ergonomics 
risk  factor  exposure  associated  with  task  and  provided  estimated  benefits  for  each 
recommendation.  (1994) 

•  For  an  automotive  parts  distribution  warehouse,  Mr.  Barker  performed  ergonomics 
assessments  of  carousel  parts  loading,  carousel  parts  picking,  small  parts  packaging  and  order 
picking  for  storage  shelving.  These  assessments  identified  the  risk  factors  present  and 
recommendations  to  reduce  exposure  to  these  risk  factors.  Mr.  Barker  also  trained 
ergonomics  team  members  in  hazard  identification  and  solution  development.  (1994) 

Product  Development  and  User  Testing 

•  For  a  check  processing  company,  Mr.  Barker  evaluated  the  impact  of  workstation  Changes  on 
the  comfort  and  performance  of  computer-based  data  entry  operators.  This  research  resulted 
in  specific  design  and  vendor  recommendations.  Mr.  Barker  developed  research  plans, 
conducted  experiments,  performed  statistical  analyses,  interpreted  data,  identified 
recommendations,  produced  reports,  and  presented  results  in  support  of  the  development  of 
image  check  processing  equipment  This  research  involved  subjective  and  performance 
measures  of  potential  products  and  benchmarking  operator  performance  on  competitors’ 
products.  Mr.  Barker  provided  product  development  support  including  participation  on 
interdisciplinary  design  teams  regarding  hardware  and  software  development.  Programs 
supported  include  software,  computer  hardware,  check  sorting  equipment  interface,  and 
manufacturing.  (Unisys,  1991). 

•  For  the  USAF  ergonomics  assessment  methodologies,  Mr.  Barker  developed  research  plans 
for  testing  the  usability  and  validity  of  the  paper  based  tools.  He  conducted  focus  groups, 
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usability  surveying,  field  testing  and  controlled  testing  in  support  of  this  research  agenda. 
(USAF,  1995-1997). 

«  Mr.  Barker  focused  on  product  design  and  end-user  methods  for  a  parts  supplier  for  a 

consumer  products  manufacturer.  He  authorized  the  work  meeting  for  producing  a  video  on 
how  to  use  their  product.  (1 994- 1 995). 

•  Mr.  Barker  conducted,  as  a  graduate  assistant,  experimental  trials  and  performed  statistical 
analyses  for  research  projects  in  discrimination  of  subjective  scaling  response  categories, 
comparative  performance  of  subjective  mental  workload  assessment  methods  and  pilot 
simulator  performance  with  varying  control  configurations.  Mr.  Barker’s  Masters  thesis  was 
conducted  on  the  potential  effects  of  the  laboratory  environment  on  the  outcome  measures  of 
software  usability  testing.  (University  of  Dayton,  1987-1989). 


Education 

Mr.  Barker  received  a  Masters  degree  in  Human  Factors  from  the  University  of  Dayton,  in  Ohio. 
Mr.  Barker  is  a  Certified  Professional  Ergonomist. 
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Biography 


Van  C.  Calvez,  CPE 

Ergonomist/Human  Factors  Engineer 

The  Joyce  Institute/A  Unit  of  Arthur  D.  Little 


Mr.  Calvez,  MS,  is  an  ergonomist/human  factors  engineer  with  The  Joyce  Institute,  a  unit  of 
Arthur  D.  Little,  within  the  Environmental,  Health,  and  Safety  Directorate.  He  provides 
consulting  and  training,  ergonomics  program  development  expertise  to  a  variety  of  industrial, 
health  care,  and  office  clients.  Mr.  Calvez  also  has  special  expertise  in  product  design  and 
traditional  human  factors. 


Human  Factors  Design 

•  Analyzed  and  developed  recommendations  for  air  traffic  controllers  as  part  of  United  States 

Air  Force  contract  for  Materiel  Command  and  Space  Command. 

♦ 

•  Analyzed  and  provided  design  input  for  command/control  centers  for  the  baggage  handling 
system  for  a  major  airport. 

•  Researched  and  developed  the  proprietary  methodology  Human  Information  Processing 
Protocol  and  applied  it  to  control/display  design,  decision  making,  error  and  cognitive  task 
analysis,  design  of  presentation  of  information  to  minimize  errors,  error-resistant  systems, 
and  error-tolerant  systems. 

•  Completed  an  extensive  literature  search  to  identify  ergonomics/human  factors  related  issues 
for  the  Air  Force. 

•  Analyzed  and  provided  input  into  the  design  of  the  workplace  to  minimize  the  impacts  of 
fatigue  in  vigilance  tasks. 


Product  Design/Human  Factors 

•  For  a  major  computer  components  manufacturer,  Mr.  Calvez  performed  a  human  factors 
evaluation  of  manufacturing  equipment  in  order  to  identify  hazards  for  catastrophic  accidents 
related  to  the  human-machine  interface.  He  used  a  task  analysis  method  based  on  human 
information  processing  capabilities  to  structure  operator  comments  and  identify  aspects  of  the 
machine  design  which  encouraged  or  permitted  operator  errors. 
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•  Mr.  Calvez  conducted  usability  tests  and  an  ergonomics  evaluation  of  a  consumer  product  to 
determine  if  the  product  concept  meets  user  requirements.  An  ergonomics  evaluation  of  the 
product  concept  was  completed  to  determine  if  the  use  of  the  product  would  contribute  to 
cumulative  stress  to  the  hand.  Usability  tests  were  examined  if  the  product  concepts  matched 
user  expectations  of  how  the  task  should  be  performed.  The  results  were  used  to  modify  the 
product  design  to  reduce  cumulative  stress  and  improve  consumer  satisfaction  with  the 
product. 


Industrial  Workplace  Consulting:  Ergonomics  Assessment  and  Design 

•  For  the  publisher  of  a  large  metropolitan  newspaper,  Mr.  Calvez  provided  a  comprehensive 
identification  and  prioritization  of  production  tasks  which  have  contributed  to  manual 
handling  injuries.  He  also  made  practical  recommendations  to  assist  the  client  in  focusing 
problem  solving  efforts,  and  worked  with  personnel  to  develop  cost-effective  solutions. 

•  At  a  components  warehouse,  Mr.  Calvez  recommended  modifications  to  a  receiving  area  to 
eliminate  awkward  lifting  postures,  unnecessary  transfers,  and  wasted  effort.  The  overall 
efficiency  of  the  operation  was  increased. 

t- 

•  For  a  munitions  manufacturer,  he  performed  ergonomics  analyses  of  machining  and  assembly 
tasks.  He  also  provided  recommendations  for  modifications  of  equipment  and  work  areas  to 
eliminate  ergonomic  hazards. 


Health  Care/Laboratory  Consulting 

•  For  a  large  hospital,  Mr.  Calvez  completed  an  ergonomics  evaluation  of  handling  tasks 
performed  within  a  surgical  processing  department.  He  provided  recommendations  for  the 
rearrangement  of  equipment  to  increase  workspace,  increase  storage  capacity,  improve 
departmental  efficiency,  and  eliminate  hazardous  handling  tasks. 

•  Mr.  Calvez  provided  a  health  care  product  manufacturing  company  with  ergonomics 
expertise  and  direction  during  the  development  of  a  training  video  for  a  patient  handling 
product.  He  provided  assistance  during  the  shooting  of  the  video  to  ensure  the  best 
ergonomic  use  of  the  product,  and  evaluated  the  resulting  video  to  ensure  that  appropriate 
procedures  were  clearly  described  and  presented. 

•  Mr.  Calvez  conducted  detailed  ergonomic  analyses  of  tasks  performed  in  a  laboratory.  He 
collected  workstation  measurements,  videotapes,  and  employee  comments;  and  completed 
elemental  task  analyses,  postural  analyses,  and  biomechanical  analyses.  Results  of  the 
analyses  and  his  recommendations  provided  the  bases  of  support  for  an  ergonomics  program. 
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Office  Workplace  Consulting:  Ergonomics  Assessment  and  Design 

•  For  a  regional  utility,  Mr.  Calvez  conducted  an  ergonomics  evaluation  to  determine  the 
potential  cause  of  ergonomic  stressors  in  the  cash  processing  department.  His 
recommendations  included  increased  lighting,  modifications  to  workarea  layout,  and  minor 
modifications  to  work  task  procedures. 

•  Mr.  Calvez  developed  specifications  for  office/computer  furniture,  chairs,  and  accessories  at 
the  corporate  offices  of  an  insurance  company.  Workstation  specifications  were  based  on 
ergonomics  principles,  relevant  tasks  performed,  and  workplace  constraints.  The  client  was 
provided  with  specific  recommendations  for  modifying  current  furniture  to  meet  ergonomics 
requirements. 


Training  Design  and  Implementation 

•  Mr.  Calvez  provided  advanced  ergonomics  training  to  engineers  at  a  printing  products 
company,  with  primary  emphasis  on  an  injury-causing  picking  task.  Excessive  disc  forces 
were  measured  as  a  part  of  the  task.  The  lifting  task  was  converted  into  a  sliding  task 
through  the  use  of  a  rake  device.  Excessive  lifting  tasks  were  eliminated  and  picking 
efficiency  was  increased. 

•  For  an  automotive  products  firm,  he  provided  advanced  ergonomics  training  for  engineers 
and  safety  representatives,  which  involved  developing  solutions  for  high  risk  tasks  and 
implementing  prototype  solutions. 

•  At  a  major  metropolitan  newspaper,  Mr.  Calvez  conducted  training  for  facilities  personnel, 
managers  and  supervisors  enabling  them  to  make  modifications  to  existing  offices,  create 
design  guidelines  and  establish  criteria  for  furniture,  accessories  and  equipment. 

•  He  trained  managers  and  supervisors  at  a  transportation  company  in  problem  identification 
techniques  through  the  use  of  an  Ergonomics  Checklist.  Upon  completion  of  training, 
supervisors  and  managers  were  able  to  provide  recommendations  for  minimizing  or  reducing 
ergonomic  stressors. 

•  For  a  major  consumer  products  manufacturing  company,  Mr.  Calvez  provided  customized 
ergonomics  training  for  both  the  manufacturing  and  office  environments.  Training  provided 
for  industrial  and  design  engineers,  safety  personnel,  facilities  personnel,  supervisors  and 
employees.  Mr.  Calvez  also  assisted  ergonomics  teams  with  measuring  the  impact  of 
changes  implemented  after  the  training. 

•  For  a  large  defense/electronics  company,  he  provided  customized  ergonomics  training  for 
representatives  of  several  divisions  of  a  large  corporation.  Provided  guidance  to  facilitate  the 
development  of  ergonomics  efforts  at  individual  sites. 
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•  For  a  financial  services  company,  Mr.  Calvez  provided  ergonomics  training  for  the  office 
environment  of  a  large  investment  firm.  Ergonomics  training  was  provided  for  facilities 
personnel,  safety,  departmental  representatives,  and  employees. 


Previous  Experience 

•  Prior  to  joining  The  Joyce  Institute,  Mr.  Calvez  was  employed  as  a  human  factors 
researcher/product  engineer  with  Fitch  Richardson  Smith/Polymer  Solutions,  Inc.  In  that 
capacity,  he  performed  ergonomics  analyses  to  specify  product  features  to  meet  the 
capabilities  and  limitations  of  the  user.  He  also  supplied  design  engineering  for  product 
components  and  assemblies,  and  provided  design-focused  user  and  market  research  for 
consumer  products,  consumer  electronics,  medical  products,  and  lawn  &  garden  products. 

He  employed  configurable  product  models  to  involve  clients  and  users  in  the  design  process. 

•  Prior  to  that,  Mr.  Calvez  was  a  project  engineer/industrial  engineer  with  Worthington  Custom 
Plastics.  He  designed  manufacturing  equipment,  hand  tools,  work  tables  and  material 
dispensers;  and  modified  product  designs  to  improve  quality  and  manufacturability.  He  also 
performed  ergonomic  evaluations  and  time/motion  studies  of  operator  workstations,  and 
implemented  improvements  to  paint  mask  operations  which  increased  worker  satisfaction  and 
substantially  reduced  cycle  times. 

•  Asa  researcher,  he  developed  a  computer-based  technique  for  obtaining  a  user's  mental 
model  of  a  technical  domain.  He  also  developed  and  tested  the  usability  of  software  and 
documentation  for  thesis  research. 

Education 

Mr.  Calvez  received  Master  of  Science  and  Bachelor  of  Science  degrees  in  Human  Factors 
Engineering  from  Wright  State  University  in  Dayton,  Ohio. 

He  is  a  member  of  the  Human  Factors  and  Ergonomics  Society  and  Tau  Beta  Pi,  the  Engineering 
Honor  Society.  He  received  a  Research  Fellowship  from  the  Office  of  Naval  Research,  and  was 
named  Outstanding  Graduate  Student  in  Human  Factors  Engineering  by  Wright  State  University. 
He  became  a  Certified  Professional  Ergonomist  (CPE)  in  1995. 

Presentations 

Calvez,  V.  "Job-Specific  Ergonomics  Training,"  VPPPA  Conference,  Washington,  DC, 
September  28, 1995. 

Calvez,  V.  “  Implementing  Ergonomics  in  the  Laboratory  Environment,”  11th  Annual 
Laboratory  Safety  and  Environmental  Conference,  Baltimore,  Maryland,  June  26-29, 1995. 

Calvez,  V.  “Ergonomics:  Overview  of  Proposed  Regulations,”  prepared  for  Gillette 
Environmental  /  Safety  Meeting,  Newport,  RI,  May  18, 1994. 


10 


Calvez,  V.  “Application  of  Methods  from  the  Proposed  ANSI  Standard  for  the  Control  of 
Cumulative  Trauma  Disorders,”  American  Industrial  Hygiene  Association  (AIHA),  Anaheim, 
CA,  May,  1994. 

Calvez,  V.  Presented  to  the  New  Jersey  Public  Service  Electric  &  Gas,  April,  1994. 

Calvez,  V.  “Applying  Ergonomics  in  the  Health  Care  Environment,”  presented  to  The  Hospital 
Employee  Health  Director’s  Association  of  Greater  New  York,  January  26, 1994. 

Calvez,  V.  “  The  New  OSHA  Ergonomics  Standard,”  presented  to  The  Hospital  Employee 
Health  Director’s  Association  of  Greater  New  York,  January  25,  1994. 

Calvez,  V.  "Ergonomic  Applications  in  the  Health  Care  Industry,"  Lenox  Hill  Hospital,  New 
York,  New  York,  July  14, 1993. 
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Biography 


Jeffrey  B.  Nelson,  CPE 

The  Joyce  Institute/A  Unit  of  Arthur  D.  Little 


Mr.  Nelson,  MSIE,  is  a  consultant  for  The  Joyce  Institute/A  Unit  of  Arthur  D.  Little 
specializing  in  industrial  and  office  consulting  and  training  and  in  the  design  of  consumer 
products  for  industrial  and  home  environments.  He  has  extensive  experience  in  office 
product  manufacturing,  automobile  assembly,  semiconductor  and  clean  room 
environments,  and  the  chemical  and  consumer  product  industries. 


Industrial  Workplace  Consulting:  Ergonomics  Assessment  and  Design 

•  For  a  modular  housing  fabricator  in  Mexico,  Mr.  Nelson  designed  a  “super  jig”  for 
use  in  the  construction  of  modular  housing,  and  evaluated  and  critiqued  the  prototype 
of  the  machine. 

•  For  a  major  hospital,  he  evaluated  the  stresses  caused  during  plasma  vial  inspection. 
His  recommendations  included  the  redesign  of  a  new  “reject”  holding  cart,  the 
adjustment  of  a  lighting  screen,  the  relocation  of  task  lighting,  the  implementation  of 
a  worker  rotation  procedure,  and  the  balancing  of  line  speeds. 

•  For  a  corrugated  box  manufacturer,  Mr.  Nelson  identified  the  stressors  placed  on 
workers  involved  in  a  box  assembly  procedure.  He  recommended  process 
improvements  and  equipment  modifications,  including  the  relocation  of  controls, 
scheduled  employee  rotation,  and  the  purchase  of  manual  lifting  devices.  He  also 
recommended  the  purchase  of  automated  folding  equipment,  and  created  a  design  for 
a  stationary  folding  apparatus. 

•  For  a  consumer  products  manufacturer,  he  evaluated  a  rotational  molding  operation, 
and  recommended  process  revisions  as  well  as  design  modifications  for  the  working 
platform,  the  material  delivery  system,  several  tools  and  molds. 

•  For  a  sewage  disposal  facility,  Mr.  Nelson  identified  the  musculoskeletal  stressors 
placed  on  workers  involved  in  the  transfer  of  motor  oil  between  containers.  He 
recommended  the  purchase  of  a  manual  siphon  pump  to  eliminate  static  loading  on 
the  shoulder,  which  was  occurring  during  the  performance  of  several  tasks. 
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•  For  a  snack  foods  manufacturer,  Mr.  Nelson  identified  the  sources  of  musculoskeletal 
stressors  placed  on  workers  involved  in  the  hand  packing  and  box  construction 
processes.  Additionally,  he  was  asked  to  identify  those  benefits  which  would  be 
gained  with  the  implementation  of  further  ergonomics  based  improvements.  Mr. 
Nelson  recommended  the  installation  of  a  mechanized  box  maker  and  label 
applicator,  the  addition  of  a  height-adjustable  table  between  two  conveyor  lines,  and 
the  height  and  weight  constraints  for  the  mylar  packaging  rolls.  He  also 
recommended  modifications  to  the  job  rotation,  and  indicated  the  potential  risk  with 
changes  in  line  speed. 

•  For  the  U.S.  Air  Force-Material  and  Space  Command,  Mr.  Nelson  served  as  the 
“Gold  Standard  Ergonomist”  during  the  task  evaluation  and  recommendations  phases. 
During  his  lengthy  visits  to  USAF  bases  in  Massachusetts  and  Florida,  he  identified 
and  ranked  those  tasks  which  contained  large  amounts  of  musculoskeletal  stressors. 
Additionally,  for  each  identified  task,  he  devised  engineering-based  solutions  to 
alleviate  those  stressors. 

•  For  the  U.S.  Postal  Service,  Mr.  Nelson  evaluated  the  mechanized  bulk  mail 
sorting/delivery  operation.  He  recommended  lighting  adjustments  for  all  tasks, 
provision  of  anti-fatigue  matting,  adjustment  of  keypad  features  for  workers  at 
standing  workstations,  creation  of  a  company- wide  job/worker  rotation  policy, 
workstation  redesign  to  facilitate  movement  and  enable  the  use  of  both  hands, 
substitution  of  manual  keypad  entry  with  a  bar-code  reading  system,  and  the  purchase 
of  headsets  and  adjustable  chairs  for  control  room  monitors.  He  also  recommended 
the  incorporation  of  advanced  ergonomics  analysis  techniques  into  future  design. 


Office  Workplace  Consulting:  Ergonomics  Assessment  and  Design 

•  For  a  regional  hospital,  Mr.  Nelson  recommended  improvements  to  the  Critical  Care, 
Histology  and  Transcription  Departments.  His  recommendations  resulted  in  the 
redesign  of  work  areas  for  enhanced  visibility,  alleviation  of  psychosocial  stresses  and 
increased  desk  space.  He  also  recommended  the  automation  of  certain  activities  to 
reduce  stresses  to  the  upper  limbs  and  remove  external  trauma;  and  recommended  the 
development  of  policy  covering  workload  distribution,  workstation  layout  and 
materials  placement. 

•  For  a  hospital,  Mr.  Nelson  redesigned  a  medical  records  storage  facility.  His 
recommendations  included  the  installation  of  new  equipment  and  the  redesign  of  out¬ 
patient  records  facility. 
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•  For  a  large  technology  firm,  Mr.  Nelson  developed  a  manual  that  instructs  computer 
input  device  users  in  how  to  improve  their  workstation  layout,  body  orientation  and 
work  habits  by  incorporating  ergonomics  principles.  For  the  manual,  he  created  a 
customized  checklist  to  reinforce  these  principles  on  an  ongoing  basis  through  the  use 
of  exercises  and  strategies  to  increase  comfort. 

•  For  a  railway  management  office  where  there  had  been  workplace  injuries,  he 
identified  stressors  and  recommended  improvements  including  the  equipment 
purchase,  institution  of  formal  evaluation  procedures,  redesign  of  work  areas  and 
suggested  design  for  future  space  planning  efforts,  training  in  office  ergonomics. 

•  Mr.  Nelson  evaluated  physical  and  psychosocial  issues  involved  with  an  aerospace- 
industry  office  worker  who  had  been  diagnosed  Fibromyalgia.  His  recommendations 
included  workstation  redesign  and  enhancements,  education  about  Fibromyalgia, 
broadening  of  the  worker’s  job  to  include  greater  responsibility  and  variety  of  tasks, 
and  training  in  office  ergonomics. 


Product  Design  Criteria  Development  for  End-User 

•  For  a  multinational  computer  design  firm,  Mr.  Nelson  developed  design  and 
operational  criteria  to  be  used  in  the  development  of  a  computer  input  device.  The 
criteria  will  allow  for  movement/control  in  all  planes  and  force  feedback  without 
causing  potential  injury. 


Training  Design  and  Implementation 

•  For  the  U.S.  Postal  Service,  Mr.  Nelson  developed  and  implemented  customized 
ergonomics  course  material,  addressing  those  ergonomics  stressors  for  tasks 
involving  sitting,  standing  and  materials  handling. 

•  Mr.  Nelson  has  trained  employees  in  ways  to  identify,  alleviate  and  prevent 
workplace  ergonomics  hazards  at  numerous  industrial  and  office  firms,  including  a 
chemical  manufacturer,  an  electrical  component/semiconductor  manufacturer,  an 
electronic  test  equipment  manufacturer,  a  hospital  equipment  manufacturer, 
newspaper  printing  and  publishing  companies,  manufacturers  of  office,  vinyl  and 
paper  products,  an  international  petrochemical  company,  a  major  pharmaceuticals 
firm,  a  pulp  and  paper  processor,  a  seafood  processor,  a  manufacturer  of  silicon 
wafers,  a  truck  manufacturer  and  a  wood  and  steel  cabinet  construction  firm. 
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Education/Professional  Activities 

Mr.  Nelson  received  a  Master  of  Science  in  Industrial  Engineering,  specializing  in 
Industrial  Ergonomics,  from  the  University  of  Cincinnati  in  Ohio.  He  holds  a  Bachelor 
of  Science  in  Kinesiology,  specializing  in  Biomechanics,  Neurology  and  Physiology  from 
the  University  of  California  at  Los  Angeles. 

He  is  a  member  of  the  Human  Factors  and  Ergonomics  Society  and  the  American  Society 
of  Safety  Engineers. 

Publications 

Nelson,  Jeffrey  B.  and  Anil  Mital,  “An  Ergonomical  Evaluation  of  the  Primary  Hand 
Flexibility  and  Capability  Changes  with  Increases  in  Examination/Surgical  Glove 
Thickness,”  Ergonomics ,  Vol.  38,  No.  4,  April  1995. 

Presentations 

Nelson,  J.  B.  and  Joyce,  M.  “Measuring  the  Results  of  Ergonomics  Training,”  ErgoCon, 

San  Jose  CA,  May  1995. 

*■ 

Nelson,  J.  B.,  and  Joyce,  M.  “Computer  Related  Injuries:  Ergonomics  and  OSHA 
Guidelines,”  Society  of  Architectural  Administrators,  Los  Angeles  CA,  May  1994. 

Nelson,  J.  B.  and  Joyce,  M.  “Ergonomics,”  Western  Safety  Congress,  Anaheim  CA,  May 
1994. 
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Biography 


Linda  Martin,  M.E.  Des. 

Human  Factors  Engineer/Ergonomist 
The  Joyce  Institute  /  A  Unit  of  Arthur  D.  Little 

Ms.  Martin  is  a  human  factors  engineer/ergonomist  with  The  Joyce  Institute,  a  Unit  of 
Arthur  D.  Little,  within  the  Environmental,  Health,  and  Safety  Directorate.  Ms.  Martin 
provides  ergonomics  consulting,  training,  and  planning  services  to  a  variety  of  industrial, 
office,  and  health  care  clients 

Ergonomics  Training  Design  and  Implementation 

Ms.  Martin  has  assisted  organizations  with  integrating  ergonomics  principles  into  design 
processes  and  management  systems.  She  has  provided  ergonomics  training  in  a  variety  of 
industries  at  all  levels;  including  frontline  workers,  engineers,  and  organizational  leaders. 
She  has  developed  and  presented  ergonomic  training  for  office/computer,  manufacturing, 
maintenance,  resource  based  industries  (forestry,  oil  &  gas  and  mining),  health  care  and 
laboratory  environments.  The  following  are  examples  of  prior  ergonomics  training 
services: 

•  Provided  customized  ergonomics  training  in  both  the  industrial  and  office 
environments  for  a  number  of  forestry  companies.  Ms.  Martin  provided  training  for 
process  engineers,  design  engineers,  EHS  personnel,  supervisors,  and  employees. 

EHS  personnel  learned  how  to  conduct  ergonomics  assessments  of  work  areas  and 
develop  short-term  solutions  based  on  ergonomics  principles.  Engineers  learned  how 
to  design  work  stations  and  manufacturing  processes  to  eliminate  ergonomic  risk 
factors  and  improve  productivity.  Supervisors  and  hourly  employees  learned  how  to 
identify  risk  factors  and  make  minor  adjustments  in  workstations  and  procedures. 

Ms.  Martin  also  assisted  ergonomics  teams  in  implementing  solutions  and  measuring 
the  impact  of  results. 

•  Developed  customized  ergonomics  training  for  a  telecommunications  company.  Ms. 
Martin  worked  with  the  client  to  developed  ergonomics  training  package  which  was 
customized  for  field  utility  applications.  Customized  content  and  illustrations  were 
based  on  ergonomic  field  studies  at  various  field  sites.  She  designed  the  training 
materials  and  train-the-trainer  modules.  She  then  trained  field  representatives  in 
conducting  ergonomics  assessments  and  developing  solutions.  She  also  provided 
them  with  the  skills  and  materials  required  to  be  internal  trainers. 

•  Developed  customized  ergonomics  training  for  a  City  Corporation  of  2 1 ,000 
employees.  Ms.  Martin  worked  with  the  client  to  developed  training  materials  which 
were  customized  to  meet  their  needs.  Illustrations  were  based  on  ergonomic  field 
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studies  at  various  production  sites.  Her  presented  the  completed  training  course  to 
managers  and  supervisors  as  well  as  a  train-the-trainer  course. 

•  Developed  a  customized  ergonomics  program  for  a  City  Corporation  and  Workers 
Compensation  Board.  Ms.  Martin  worked  with  the  clients  to  strategically  plan  a 
ergonomics  plan,  budget  process  and  roll  out  package  for  the  organizations. 

Ergonomics  Assessment  and  Design 

Ms.  Martin  has  provided  ergonomics  consulting  for  a  variety  of  clients  in  resource  based 
industries  specifically  forestry,  health  care,  laboratory,  and  office  environments.  Much  of 
this  work  has  focused  on  improving  process  efficiency,  improving  product  quality,  and 
preventing  work-related  fatigue  and  musculoskeletal  disorders  (such  as  back-injuries, 
carpal  tunnel  syndrome).  Ms.  Martin’s  work  ranges  from  identifying  and  prioritizing 
ergonomic  hazards  in  entire  facilities  to  providing  engineering  assistance  with  the 
development  of  new  tools  and  equipment.  The  following  are  some  examples  of  this  type 
of  work  in  three  major  areas:  industrial,  health  care/laboratory,  and  office  workplaces. 

Industrial  Workplace  Consulting 

The  following  are  examples  of  ergonomics  consulting  activities  in  resource  based 
industries  environments: 

•  Recommended  equipment  and  process  modifications  for  natural  gas  company  in  the 
meter  repair  and  shipping  and  receiving  to  eliminate  repetitive  motions,  heavy  lifting 
and  examine  flow  patterns.  By  rearranging  the  receiving  department  and  manual 
handling  equipment  it  was  possible  to  eliminate  a  large  amount  of  unnecessary 
manual  handling 

•  Identified  and  prioritized  high  risk  production  tasks  for  a  lumbermill.  Ms.  Martin 
reviewed  existing  injury  records  and  conducted  an  ergonomics  assessment  of  all 
production  activities.  She  then  made  recommendations  to  assist  the  client  in  focusing 
problem  solving  efforts,  and  worked  with  personnel  to  develop  cost-effective 
solutions. 

•  Performed  ergonomics  analyses  of  saw  filing  shop,  wrap  and  strapper  and  handling 
system  in  a  lumbermill.  She  also  provided  recommendations  for  modifications  of 
equipment  and  work  areas  to  eliminate  ergonomic  hazards. 

Office  Workplace  Consulting 

The  following  are  examples  of  ergonomics  consulting  activities  for  office/computer  tasks: 

•  Developed  specifications  for  office/computer  furniture,  chairs,  and  accessories  at  the 
corporate  offices  of  an  Telecommunications  company.  Workstation  specifications 
were  based  on  ergonomics  principles,  relevant  tasks  performed,  and  workplace 
constraints.  The  client  was  provided  with  specific  recommendations  for  modifying 
current  furniture  to  meet  ergonomics  requirements. 
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•  Conducted  an  ergonomics  evaluation  to  determine  the  potential  cause  of  ergonomic 
stressors  in  the  cash  processing  department  for  a  regional  utility.  She 
recommendations  included  increased  lighting,  modifications  to  work  area  layout,  and 
minor  modifications  to  work  task  procedures. 

Health  Care/Laboratory  Consulting 

The  following  are  examples  of  ergonomics  consulting  activities  in  health  care 
environments: 

•  Completed  an  ergonomics  evaluation  of  handling  tasks  performed  within  a  surgical 
processing  department  for  a  large  medical  center.  Ms.  Martin  provided 
recommendations  for  the  rearrangement  of  equipment  to  increase  workspace,  increase 
storage  capacity,  improve  departmental  efficiency,  and  eliminate  hazardous  handling 
tasks. 

•  Conducted  detailed  ergonomic  analyses  of  tasks  performed  in  a  laundry  and  kitchen 
service  within  a  hospital.  She  collected  workstation  measurements,  videotapes,  and 
employee  comments;  and  completed  elemental  task  analyses,  postural  analyses,  and 
biomechanical  analyses.  Results  of  the  analyses  and  her  recommendations  provided 
the  bases  of  support  for  an  ergonomics  program. 

Education 

Ms.  Martin  received  Master  of  Environmental  Design  from  the  University  of  Calgary  and 
Bachelor  of  Science  degrees  in  Rehabilitation  Medicine  from  University  of  Alberta.  She 
is  presently  working  on  a  Ph.D.  in  Environmental  Design  at  the  University  of  Calgary. 

She  is  a  full  member  of  the  Human  Factors  Association  of  Canada  and  the  Canadian 
Association  of  Occupational  Therapists.  She  was  runner  up  for  the  Gold  Medal  at  the 
University  of  Calgary  for  her  graduate  research  work. 

Partial  Client  List 

Telus  Communications  Inc. 

Ranger 

Northwestern  Utilities  Ltd. 

Canadian  Western  Natural  Gas 
Altasteel 
Hewlett  Packard 
Capital  Health  Authority 
City  of  Calgary 
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Biography 


Andrew  J.  Marcotte,  MS,  CPE 

Mr.  Marcotte  is  the  Assistant  Director  of  Ergonomics  for  The  Joyce  Institute  unit  of 
Arthur  D.  Little.  Mr.  Marcotte  is  responsible  for  planning,  managing,  and  providing 
technical  direction  and  on-going  support  for  large  ergonomics  projects.  His  current  work 
involves  helping  clients  identify  innovative  approaches  to  implementing  ergonomics 
programs  including  priority-based  worksite  analysis,  practical  solution  design,  and 
ergonomics  specification  development  for  future  installations/purchases,  which  generate 
results  that  show  measurable  improvements  in  employee  health  and  safety,  workers’ 
compensation  costs  as  well  as  company  performance  and  profitability.  Together  with  the 
ergonomics  consulting  staff,  Mr.  Marcotte  has  designed  a  series  of  training  courses  for 
addressing  ergonomics  concerns  in  industrial,  maintenance  and  service,  administrative, 
laboratory,  and  health  care  environments  as  well  as  integrating  ergonomics  into  the 
product  development  cycle. 

Relevant  Professional  Experience 

•  Mr.  Marcotte  served  as  the  technical  project  manager  for  a  $  1 .2  MM  contract  with 
the  United  States  Air  Force.  Project  goals  included  developing  a  systematic  approach 
to  worksite  analysis  and  hazard  prevention  and  control  and  implementing  that  system 
as  a  model  for  continued  application  by  Air  Force  personnel.  The  system  uses  a 
multi-tiered  approach  to  problem  identification  that  identifies  and  prioritizes  potential 
problem  departments  based  on  past  injury  illness  reports  and  a  Job  Requirements  and 
Physical  Demands  Survey.  The  next  step  uses  a  risk  factor  checklist  to  identify 
problem-solving  priorities  for  jobs  within  the  problem  departments.  Sixty  solution 
design  Case  Studies  have  been  created  to  address  common  ergonomics  problems 
associated  with  (aircraft  and  systems)  maintenance  and  inspection,  assembly, 
warehousing,  and  administrative  work  tasks.  Mr.  Marcotte’s  project  responsibilities 
included  developing  the  project  workplan,  developing  the  models  for  data  collection, 
Survey,  Checklist,  and  Case  Study  design,  drafting  ongoing  status  and  the  final 
project  reports,  and  acting  as  a  technical  liaison  with  Air  Force  program  managers 
during  implementation  and  program  success  measurement. 

•  Mr.  Marcotte  developed  ergonomics  program  Guidelines  for  a  large  aerospace 
conglomerate.  Mr.  Marcotte,  integrated  ergonomics  into  existing  corporate  safety  and 
health  policy,  developed  a  Guidance  Document  which  identifies  individual  company 
responsibilities  and  drafted  a  “model”  ergonomics  implementation  strategy  which 
could  be  used  throughout  the  organization  as  the  basis  for  site-specific  program 
planning  and  implementation. 
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•  Mr.  Marcotte’s  work  with  a  national  office  products  manufacturer  has  resulted  in  1 .7 
million  in  cost  savings  over  a  2.5  year  period.  Mr.  Marcotte  was  responsible 
establishing  the  original  strategy  document  (written  ergonomics  plan)  which  specified 
the  systematic  integration  of  ergonomics  analysis  and  design  methodologies  into  the 
organizations  day  to  day  operations  and  continuous  improvement  processes.  His 
other  responsibilities  included:  conducting  worksite  analysis  and  establishing  initial 
priorities  for  improvement;  identifying  short-  and  long-term  solutions  to  priority 
ergonomics  problems;  providing  all  technical  training  for  committees  and  personnel 
who  had  impact  on  the  way  that  jobs  and  the  workplace  are  designed;  measuring 
results;  and  working  with  the  company  to  develop  “best  practices”  that  are  now  used 
as  technology  is  transferred  throughout  the  organization. 

•  Mr.  Marcotte  provided  technical  expertise  to  a  large  aerospace/fuel  systems  company. 
His  expertise  was  used  to  ensure  that  a  change  to  cell  manufacturing  technology 
would  minimize  employee  exposure  to  ergonomics  hazards,  improve  material  flow, 
and  maximize  cell  productivity.  Mr.  Marcotte  also  used  the  recommendations  (e.g., 
analysis  results,  priority  list,  solutions  and  implementation  strategy)  to  train  engineers 
and  manufacturing  personnel  to  enable  them  to  address  other  common  ergonomics 
issues  throughout  the  facility. 

•  Mr.  Marcotte  performed  a  series  of  workplace  assessments  at  the  request  of  a  missile 
systems  manufacturer.  The  ergonomics  assessment  was  prompted  by  union  officials 
due  to  past  employee  complaints  and  a  concern  about  cumulative  trauma  injuries. 

The  assessment  results  were  used  by  the  union  and  management  to  identify  a  list  of 
action  priorities  for  implementing  improvements  throughout  the  facility.  Processes 
included  wiring  harness  assembly,  soldering/bench  work,  metal  fabrication, 
component  assembly,  visual  inspection,  and  heavy  material  handling. 

•  Mr.  Marcotte  provided  technical  review  to  a  major  helicopter  manufacturer,  Bell 
Helicopter,  to  maximize  the  effectiveness  of  solutions  provided  for  all  primary  work 
functions  associated  with  helicopter  assembly.  He  also  developed  the  company’s 
ergonomics  training  manual  and  course  structure  for  a  Self-Paced  supervisor  and 
employee  seminar  which  is  used  on  an  on-going  basis. 

•  Mr.  Marcotte  developed  ergonomics  program  implementation  strategy  and  provide 
on-going  technical  guidance  to  engineers,  project  managers,  and  ergonomics 
committees  for  a  major  manufacturer  of  aircraft  engines.  The  work  involved 
conducting  worksite  analysis  to  identify  department/plant  priorities  and  developing 
solutions  which  can  be  implemented  in  the  short-term.  Whenever  possible  solutions 
are  designed  to  improve  employee/work  area  performance  in  addition  to  minimizing 
employee  exposure  to  ergonomics  risk  factors.  Currently  serving  as  the  Ergonomics 
Expert/Resource  for  the  aircraft  engine  manufacturing  and  testing  complex  and  ad  hoc 
member  of  several  of  the  most  active  ergonomics  committees.  Projects  have  included 
developing  a  material  handling  device  to  transport  aircraft  engines  and  build  fixtures, 
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evaluating  proposed  work  area/process  redesign  for  the  primary  packaging  and 
distribution  center,  developing  work  area  layout  for  a  major  administrative  area. 

•  Mr.  Marcotte  has  developed  work  area  and  workstation  design  specifications  for  an 
international  law  firm.  The  specifications  are  used  to  communicate  job-specific  needs 
to  architects  as  they  design  workplaces  to  accommodate  legal  secretaries,  paralegals, 
attorneys,  information  services  staff,  data  processing,  and  other  administrative  support 
personnel.  Based  on  is  previous  work  with  the  firm,  Mr.  Marcotte  provides  on-going 
ergonomics  technical  review  of  office  relocation  and  re-design  efforts. 

•  As  an  employee  of  General  Motors,  Central  Foundry  Division,  Mr.  Marcotte  provided 
technical  ergonomics  design  expertise  for  the  development  of  new  foundry  process 
technology.  As  a  former  production  supervisor  and  development  engineer,  Mr. 
Marcotte’s  projects  (e.g.,  materials  handling,  work  area  layout)  have  generated 
increases  in  molding  productivity  by  33  percent. 

Education/Professional  Activities 

Mr.  Marcotte,  a  certified  professional  ergonomist,  earned  his  Master  of  Science  in 
Industrial  and  Operations  Engineering,  specializing  in  Ergonomics/Occupational  Health 
&  Safety  Engineering,  from  the  University  of  Michigan  at  Ann  Arbor.  He  holds  a 
Bachelor  of  Science  in  Industrial  and  Systems  Engineering  from  the  General  Motors 
Institute  at  Flint,  Michigan. 

He  previously  served  on  the  ANSI  -  Z-365  Subcommittee  and  is  a  member  of  the  Human 
Factors  and  Ergonomics  Society,  the  American  Society  of  Safety  Engineers  and  the 
Institute  of  Industrial  Engineers. 

Research/Publications 

Marcotte,  A.  J.,  Barker,  R.  T.  Level  I  Ergonomics  Assessment  Methodology  for  Maintenance  and 
Inspection  Work  Areas,  United  States  Air  Force  Materiel  Command/Armstrong  Laboratories,  December 
1996. 

Marcotte,  A.  J.,  Barker,  R.  T.  Level  I  Ergonomics  Assessment  Methodology  for 
Administrative  Work  Areas,  United  States  Air  Force  Materiel  Command/Armstrong 
Laboratories,  October  1996. 

Marcotte,  A.  J.,  Barker,  R.  T.  Job  Requirements/Physical  Demands  Survey  Methodology,  United  States  Air 
Force  Space  Command/Armstrong  Laboratories,  September  1996. 

"Ergonomics  Applied  to  Product  &  Process  Design  Achieves  Immediate,  Measurable  Cost  Savings," 

HFES,  San  Diego,  CA,  October,  1995. 

Marcotte,  A.  J.,  Weinkamp,  P.,  Jackson,  P.,  and  Bretz,  T.,  “Moving  Mattresses  Safely:  A  Case  Study  in 
Manual  Materials  Handling,”  International  Ergonomics  Association,  Warsaw,  Poland,  June  1993. 

Marcotte,  Andrew  and  Adams,  Edie,  “A  Manager’s  Guide  to  Ergonomics  in  the  Chemical  and  Allied 
Industries,”  Chemical  Manufacturers  Association  (CMA),  1992. 
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Marcotte,  A.  J.,  “Implementing  a  Successful  Ergonomics  Program  Using  In-House  Expertise:  Results 
Through  Training,  “  Proceedings  of  the  IEA  Annual  Meeting,  Paris,  France,  July,  1991. 

Marcotte,  A.  J.  and  Martino,  John,  “Injury  Prevention  In  An  Incentive-based  Production  Environment:  A 
Case  Study/’  Proceedings  of  the  IFIESR  Annual  Meeting,  Lake  Tahoe,  Nevada,  June  1991. 

Cook,  Robert  E.  and  Marcotte,  A.  J.,  “Ergonomic  Improvement  in  Games  Manufacturing:  A  Case  Study,” 
Proceedings  of  the  HFS  34th  Annual  Meeting,  Orlando,  Florida,  October  1990. 

Presentations 

Marcotte,  A  .J.,  "Making  a  Business  Case  for  Ergonomics,"  Organization  Resources  Councelors,  Inc.,  San 
Francisco,  CA,  September,  1995. 

Marcotte,  A.J.,  "Ergonomics:  Value-Added  Safety,"  Fisher  Scientific  Conference,  Two  Rivers,  WI,  March, 
1995. 

Marcotte,  A.J.,  "Ergonomics  Workstation  Analysis  for  the  Bedding  Industry,"  International  Sleeps 
Products  Association,  Albuquerque,  NM,  March,  1995. 

Marcotte,  A.J.,  "How  to  Develop  a  Practical  Written  Ergonomics  Plan,"  Manufacturers'  Education 

Council,  Columbus,  OH,  February,  1995. 

*■ 

Marcotte,  A.  J.,  “Ergonomics  Measurable  Results  in  the  Chemical  Industry:  Case  Studies.”  Presentation 
identifying  key  applications  and  results  of  ergonomics  improvements  in  chemical  process  operations. 
National  Safety  Congress,  Chicago,  IL.  October  1993. 

Marcotte,  A.  J.,  “Public  Utilities  and  Ergonomics:  Problems  and  Solutions.”  Case  study  presentation 
identifying  potential  scope  of  problems  and  results  of  improvements  in  public  utilities  work.  National 
Safety  Congress,  Chicago,  IL.  October  1993. 
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ANSWERING  THE  CHECKLIST  QUESTIONS 
The  wording  of  the  questions  is  clear 

The  wording  of  the  response  choices  is  clear 

The  risk  criteria  are  well  defined  within  each  question 

The  risk  criteria  are  easily  estimated  without  measurement 

I  encountered  workplace  ergonomics  concerns  that  were  not  covered  by  the  checklist 

The  tool  has  sufficient  breadth  of  risk  identifiers 

i  am  reasonably  certain  that  most  of  my  responses  are  correct 


USING  THE  MATRICES  TO  IDENTIFY  SOLUTIONS 
The  descriptions  of  the  recommendations  are  clear 

The  tool  provides  a  sufficient  variety  of  solutions 

The  tool  is  useful  for  generating  solutions 

The  tool  is  better  suited  to  an  ergonomist  that  a  non-ergonomist 

I  could  not  find  solutions  for  concern  that  I  identified 

The  tool  helps  me  to  generate  more  solution  options  than  I  would  on  my  own 

The  tool  helps  me  to  generate  better  solution  options  than  I  would  on  my  own 


OVERALL  USE  OF  THE  GUIDE 

The  tool  is  well  suited  for  field  use 

The  tool  is  useful  for  ranking  jobs 


The  scores  can  be  generated  quickly  and  easily 

The  activities  performed  in  assembly  and  warehouse  are  covered  in  the  tool 

The  scores  can  be  clearly  interpreted 

The  written  instructions  were  easily  understood  and  complete 
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Strongly  Disagre 


How  useful  do  you  think  this  guide  will  be  for  identifying  corrective  actions? 


What  aspects  of  the  guide  did  you  find  the  most  useful? 


Were  there  any  aspects  of  the  guide  that  you  found  difficult  or  awkward  to  use? 
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ANSWERING  THE  CHECKLIST  QUESTIONS _  1.928571 

The  wording  of  the  questions  is  clear  1.75 


@030 


The  tool  is  useful  for  ranking  jobs 
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Were  there  any  aspects  of  the  guide  that  you  found  difficult  or  awkward  to  use? 
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O'.,?  ■  . 

Scenario 

Base  ■ 

Tasks  '■■MU ■■  ■  ■  ■  . 

■ 

Prepare  and  Inspect 
Oxygen  Mask 

Eglin 

Prepare  &  Inspect  Life  Support 
Equipment  8  hours/day 

2 

Cashiers 

WP 

Cashier  8  hours/day 

3 

Pack  &  Crate 

Peterson 

Packing/Shipping  6  hours/day 
Driving  2  hours/day 

4 

Lubricating  Trailers 

Hill 

Lubricate  2  hours/day 

Bi 

Cooking 

Hill 

Cooking  8  hours/day 

Stock  Commissary 

WP 

Stocking  8  hours/day 

7 

Food  Service 

Hill 

Food  Serving  6  hours/day 

8 

Warehouse 

W-R 

Packing/Shipping  6  hours/day 

9 

Meat  Cutting 

Hill 

Meat  Cutting  8  hours/day 

10 

Dishwashing 

Eglin 

Dishwashing  8  hours/day 

